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Executive Summary 


Introduction 

Research support for Virtual Environment technology development has been a part 
of NASA’s human factors research program since 1985. Under the auspices of the 
Office of Aeronautics and Space Technology (OAST), initial funding was provided to 
the Aerospace Human Factors Research Division, Ames Research Center, which 
resulted in the origination of this technology. Since 1985, other Centers have begun 
using and developing this technology. At each research and space flight center, 
NASA missions have been major drivers of the technology. 

This White Paper was the joint effort of all the Centers which have been involved in 
the development of technology and its applications to their unique missions. 

Appendix A is the list of those who have worked to prepare the document, directed 
by Dr. Cynthia H. Null, Ames Research Center, and Dr. James P. Jenkins, NASA 
Headquarters. 

This White Paper describes the technology and its applications in NASA Centers 
(Chapters 1 , 2 and 3), the potential roles it can take in NASA (Chapters 4 and 5), 
and a roadmap of the next 5 years (FY 1994-1998). The audience for this White 
Paper consists of managers, engineers, scientists and the general public with an 
interest in Virtual Environment technology. Those who read the paper will determine 
whethe r t his roadmap, or others, are to be followed. 

Summary of the Technology 

"Virtual reality is the human experience of perceiving and interacting through 
sensors and effectors with a synthetic (simulated) environment, and with simulated 
objects in it, as if they were real" (Virtual Reality Technology Report to the Office of 
Science and Technology Policy, Executive Office of the President). 

Virtual reality is a unique method to achieve this simulation because of its capability 
to immerse and envelop the human user in the simulated environment. This 
definition engages the human experience and human interaction such that the 
human performance that results from immersion in the virtual reality benefits the 
human user. The technology needed to achieve virtual reality is called Virtual 
Environment Technology. 
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Virtual Environment displays are interactive, computer-graphics based, head- 
referenced displays that create the illusion that their users are in a place other than 
where they actually are. This illusion is created through the operation of three basic 
types of equipment: 1) sensors to detect human action, such as a head-mounted 6 
degree of freedom position sensor; 2) effectors to influence the operators' senses, 
such as a stereoscopic display; and 3) special purpose hardware to link the output of 
the sensors to inputs for the effectors so that they may produce sensory effects 
resembling those experienced by inhabitants of a physical environment. In a Virtual 
Environment this linkage is accomplished by a simulation computer. In a head- 
mounted teleoperator display--a display closely related to a Virtual Environment 
display--the linkage is accomplished by the robot manipulators, vehicles, control 
systems, sensors and cameras at a remote work site. 

These displays potentially provide a new communication medium for human- 
machine interaction which will be cheaper, more convenient, and more efficient than 
former interface technologies. In teleoperation or planetary surface visualization 
applications, for example, Virtual Environments can provide techniques for solving 
problems caused by long transport delays or inability to place remote cameras in 
optimal viewing positions. Additionally, the totally synthetic character of computer 
graphics based Virtual Environments allows the introduction of symbolic, geometric, 
and dynamic enhancements that can enable visualization and interaction modes that 
are totally unrealizable in physical environments. 

Virtual Environment technology is still in its infancy. However, there is great 
potential for this technology for NASA. Several NASA centers, following initial 
research and development at AMES in the Human Factors Division, are now 
investigating and developing Virtual Environment technology for specific NASA tasks 
and missions. The following outlines the different responsibilities by Center. 

Center Activities in Virtual Environment 

Ames Research Center 

• Responsible for human performance research relevant to developing Virtual 

Environment for NASA applications. 

• Responsible for the development of human centered technology for aeronautics. 
Goddard Space Flight Center 

• Responsible for unmanned scientific studies and applications for unmanned 

space flight, in the areas of: 

Space physics 
Astrophysics 
Earth sciences 
Flight project support 

Jet Propulsion Laboratory 

• Responsible for research, development and applications for unmanned 

spacecraft, satellites and ground data systems. 
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Johnson Space Flight Center 

• Responsible for manned space flight research, development, and applications. 

• Responsible for astronaut training. 

Marshall Space Flight Center 

• Responsible for spacecraft design, structure, development and operations. 

These centers have worked together to evaluate the current state of the technology, 
plan for future activities and to organize this report. This group makes the following 
comments and conclusions on Virtual Environment within NASA. 

Conclusions 

Since beginning research and technology development in 1985, NASA Centers have 
learned important lessons about the technology itself and the value it can provide in 
accomplishing the gamut of NASA’s missions in aeronautics, science, and space. 

1. Cost savings could be dramatic since Virtual Environment can potentially allow 

change to be made in a small way which can have a large effect; can 
potentially analyze situations with Virtual Environment with capabilities not 
heretofore available; can potentially analyze situations quicker and cheaper 
than with conventional methods; analyses can potentially be done which allow 
unique insights for investigators/scientists. 

2. Networking is critically important to users of Virtual Environment because of the 

need to share data among many investigators. 

3. Since model and database development are critical and time consuming for 

virtual world development, techniques for streaming this modeling are 
essential. Standardization, and maintenance are also critical and need to be 
address. 

4. NASA recognizes the need for human performance validation and that human 

performance requirements drive the technology. 

5. The productivity benefits of Virtual Environment will critically depend upon 

validated modeling of the specific task domain. 

6. Challenging mission applications within NASA call for a responsive Virtual 

Environment technology. Typically, this is a high technology need. NASA 
has a leadership role in the technology development without depending upon 
the value of low-tech commercial development. 

7. Virtual Environment is pervasive and the implications are extensive within 

NASA's many missions and research programs. NASA should be prepared 
to respond to such demands by supporting the technology. 
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8 Uses of Virtual Environment technology for human performance applied Judies 
and critical descriptive research matches both applied mission needs as well 
as fundamental research needs. 


9 Uses of Virtual Environment technology provide a flexible, relatively low-cost 

method for operational analysis, scientific studies, and cntical discipline 

research. 

10 Althouqh Virtual Environment technology is evolutionary, building upon 

technologies such as simulation, computer graphics and so forth; implications 

for its use are revolutionary. 

11 A well-documented, international interest and economic position of Virtual 

Environment technology exists. NASA has a well understood role in 
technology development and transfer. This transfer must be fostered if t e 
US is to maintain its leadership position. 

1? Current Virtual Environment systems generally do not have sufficient sensory- 
S and human-machine interface design to deliver the performance 

necessary to achieve many of the above potential applications but 
foreseeable technical advances may change this situation within 1-3 years. 
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Detailed 5-year Virtual Environment Technology 

Plan 

The Outline which follows is a summary of the research at each Center which had 
FY 94 funding, as of April 1993, and plans for additional research which was 
"Unfunded" as of that date. 
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Unfunded 

Milestone 



• Testing of integrated optimizer with proximity 
operations planner. 



• Virtual teleconferencing for Cassini MOS. 

• Virtual S/C design for early integration & test. 





• "Transportable" VE system. 

• Stowage virtual Spacelab module. 

• Incorporate anthropometric model. 

• Incorporate physical model. 

• 1GDS/EMS (Integraph) CAD file 
translation/conversion. 

• HFE for IML-2, SSFP WPO 1/Payloads. 
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Funded 

Milestone 

• Complete part-task perceptual studies at Ames 
to measure the enhanced speech intelligibility 
obtained with the spatial synthesis methods 
used in the communications prototype at KSC 
(funded by Ames DDF). 

• Complete prototype hardware for simulation of 
reverberant environments based on data- 
reduction techniques applied to HRTFs (PCA 
model). 


• Complete study of interactive, virtual windows 
for mission ops. 

• Application of AI techniques to VEs. 



• Macro-ergonomics: "real" and virtual control 
rooms (POCC,DCR). 

• Macro-ergonomics analyses of POIC/PCA 
layout/features. 

• Sensing And force-reflecting exoskelton 
(SAFiRE): hand. 

| • Interactive virtual panels. 
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• Establish requirements for response-speed and 
visual fidelity based on field studies in space- 
analog envimoments and simulation. 

• Complete prototype hardware for large-scale 
simulation of reverberant environments. 

• Introduction of Navie planning tool into 
operations center. 

• Virtual DSOT cockpit. 

• Use of autonomous agents in mission ops. 

• Sensing And force-reflecting exoskelton 
(SAFiRE): arm. 

* IGDS/EMS (Integraph) CAD File immersion. 

* Enhanced virtual USML-2. 

• USML-2 Mission Support. 

• HFE for USML-2, SSFP WPOl/Payloads. 

• Continue applications development/ 
demonstration/validation. 

• Continue application driven system 
enhancements. 

• SL3 DDM unplanned IFM ’Virtualization". 



• Psychoacoustically evaluate prototype 
hardware for large-scale simulation of 
reverberant environments. 

• Demonstrate VE-based remote fault diagnosis. 

• Complete interactive prox ops optimizer 
command syntex. 
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• Psychoacoustically evaluate prototype hardware 
for simulation of reverberant environments with 
PCA reduced HRTFs. 
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• Redesign Convolvotron (Mark II) to take full 
advantage of PCA-based modeling, enable real 
time rendering of complex environments. 
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FY 97 GSFC • Distributed, parallelized VE-FAST with medium 
resolution and full color. 
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Develop rings dynamics model for Saturn. 

Add time dependent and frequency dependent 
access for interactive planetary atlas. 

Complete global maps of Saturn satellite; use 
Galileo data to augment satellites of Jupiter. 

Complete development of visualization tools 
to rapidly access and Interactively explore 
massive, n-dimensional data bases. 

Complete interactive, manipulative techniques 
& tools for complex scientific models & data. 

Develop visualization for energy flux for all 
spacecraft constructed after 1987. 
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VE Control for Teleoperations/Telerobotics 



FY96 JPL • Develop interface strategies to compensate for 

long round trip light times. 



Training Systems 


NASAVE R&T 


Page 18 


Unfunded 

Milestone 

• VE developed to train cable layout & 
assembly task. 

• Spacecraft integration training in a VE. 





• Proof-of-concept demonstration of VE for 
shuttle/station training for two astronauts. 

• Real-time routine sharing of VE between 
three to five distant sites. 

* Temperature/tactile/force feedback. 
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• Familiarization training virtual spaceiao 
module. 

• Science/payload system Vb training. 

• USML-2 Familiarization training in virtual 
spacelab module. 

• SAFiRE/lML-2 glovebox ops trainer. 

♦ IML-2 virtual panels trainer. 
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Milestone 



• Proof-of-concept demonstration of VE for 
shuttle/station training for one astronaut. 

• Real-time, routine sharing of VE between two 
distant sites. 

• Temperature/tactile feedback. 

_____ ■ — — " — -■ 

• Initiate development of tools for building VE- 
based training svstems. 
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coaching and aiding. 


Chapter 1 : Introduction 

Virtual Environments: Definition 

What is a virtual environment? 

Virtual environment (VE) displays are interactive, computer-graphics based, head-referenced displays 
that create the illusion that their users are in a place other than where they actually are. This illusion is 
created through the operation of three basic types of equipment: 1) sensors to detect human action, such 
as a head-mounted 6 degree of freedom position sensor; 2) effectors to influence the operators' senses, 
such as a stereoscopic display; and 3) special purpose hardware to link the output of the sensors to 
inputs for the effectors so that they may produce sensory effects resembling those experienced by 
inhabitants of a physical environment. In a virtual environment this linkage is accomplished by a 
simulation computer. In a head-mounted teleoperator display--a display closely related to a virtual 
environment display-the linkage is accomplished by the robot manipulators, vehicles, control systems, 
sensors and cameras at a remote work site. A number of different names have been used to describe 
virtual environment research. Some like the oxymoronic “artificial reality" or "virtual reality" suggest much 
higher performance than the current technology can generally provide. Others like "cyberspace" are 
puzzling neologisms not closely related to the meaning of their linguistic roots. Terms like "virtual worlds" 
or "virtual environment" seem preferable since they are linguistically conservative and may be related to 
existing well established terms such as a virtual image (Ellis, 1991). 

Why are virtual environments useful? 

Virtual environments are communications media 

These displays potentially provide a new communication medium for human-machine interaction which 
will be cheaper, more convenient, and more efficient than former interface technologies. In teleoperation 
or planetary surface visualization applications, for example, virtual environments can provide techniques 
for solving problems caused by long transport delays or inability to place remote cameras in optimal 
viewing positions. Additionally, the totally synthetic character of computer graphics based virtual 
environments allows the introduction of symbolic, geometric, and dynamic enhancements that can enable 
visualization and interaction modes that are totally unrealizable in physical environments. 

Communications media have multiple uses. 

Since virtual environment display systems amount to communications media, they are intrinsically 
applicable to practically anything; education, procedure training, teleoperation, high-level programming, 
remote planetary surface exploration, exploratory data analysis, and scientific visualization. One unique 
feature of the medium, however, is that it enables multiple, coordinated, real-time foci of control in an 
environment. Tasks that involve manipulation of objects in complex visual environments and also require 
frequent, concurrent changes in viewing position, for example, laparoscopic surgery (Green, Satava, 
John-Hill, & Simon, 1992) are tasks that are naturally suited for virtual environment displays. Other tasks 
that may be mapped into this format are also may uniquely benefit. 

How are virtual environments made? 

The display technology works by developing a real-time, interactive, personal simulation (Foley, 1987) of 
the content, geometry, and dynamics of a work environment directly analogous to that used for traditional 
vehicle simulation (Cardullo, 1993; Rolfe, & Staples, 1986). But the unlike vehicle simulation, typical 
virtual environment simulation is unmediated. The users themselves are in an environment, not in a 
vehicle which is in an environment, and the hardware producing the simulation is more often than not, 
worn and not entered. The definition of a virtual environment requires three distinct operations. First, the 
shape and kinematics of the actors and object needs to be specified via a modeling program. Second, 
the modes and rules of interactions of all actors and objects need to be established for all possible 
interactions among them and with the environment itself. Third, the extent and character of the 
enveloping environment needs to be specified. 
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Where is virtuel environment reseerch end development conducted? 

Precursors in telerobotics and computer graphics _ 

Components of virtual environment display technology have been under e 19 % . 

1960 Philco and Argonne National Laboratory work on telepresence displays (Comeau, & Brian, 96 , 
Goertz 19M, Goeitz. Mingesz, Potts, & Lindberg, 1965). More ^recent wo* .has beer ‘assoaat^ with the 
development of computer graphics systems through the pioneering work of Ivan Sutheriand (Myers, & 
Sutherland, 1968; Sutherland, 1965; Sutherland, 1970) at Harvard and Utah. As an outgrowth of the 
association with computer graphics development, virtual environment development has been most 
intensively pursued by aircraft simulation groups (CAE Electronics, 1991; British Aerospace, 1993) 
interested in alternative display systems to expensive projection dome systems. Most recently, interest 
Sal simSam provided by virtual environment displays has spread Mo telerobobos scientite i date 
visualization, planetary surtace exploration, video game development, and interactive art (See Pimentel. S 
Teixeira. 1993, for a review). 

Because^ the broad potential applicability, a number of NASA centers have followed NASA Ames lead, 
in 1985 in the Aerospace Human Factors Research Division at Ames, Michael McGreevy and James 
Humphries assembled the first low cost virtual environment system. Many of the P r °9rams following this 
initialdevelopment have been pursued under the aegis of a number of different programmatic titles, ■ lor 
example teleoperations, telerobotics, applied computer graphics, and scientific visualization. Since the 
display technology is potentially the quintessential technique for scientific investigation of ^ny 
psychophysical, physiological, human factors and perceptual questions, many biological, Physiologrcal 
and coanitive scientists are interested in the technology as a new tool for their research. In fact, research 

much uselul design inlormalion lor Ihe engineering of virtual environment 

displays (Ellis, Kaiser, and Grunwald, 1993). 

Who conducts virtual environment research and development? 

ScienUsts^nd developers and those with nonprofessional interests in virtual environment technology may 
be divided in two general groups. Those who wish to use the technology to advance their particular 
profession or interest and those who wish to develop and perfect the technology itself. One mig 
S a Sine biologist interesting in catching jellyfish at great depths with a human i lad to. specialist 
who wish to improve the design of an interface for route planning and operation of undersea 
vehicles. 

The distinction between these two groups is not always clear as many of the users often have 
overestimated the actual capabilities of existing systems. Though they may consider themselves to lb 
users they are actually developers who need to significantly improve the technology for their specific 
tasks Un?ortuna?elJ . because their expertise is primarily in a task domain, they are unaware of the man- 
machine interface principles needed to select and integrate appropriate equipment to enable them to 
efficientlv achieve practical goals. Consequently, the product from this kind of development may be a 
"conceptual demo' ,P which suggests possible applications but which itself is not practically t useful. For 
exampte the field use of a helmet mounted display developed for remote teleoperation of an underwater 
vehicle to be used for research in Antarctica proved difficult due to the limitations in the visual quality o 
the images it could present (Stoker, 1993). 

nontechnical groups and organizations without specific expertise nf fh ip nroups 

scientific issues eg the Meckler Foundation, and the Education Foundation. Some of th e se 9 r °ups f 

!S ThfvSte^hiScal SSSntX m5p fe underscored b, a 

where there are no experts, and everyone can be one! 


NASA VE R&T 


23 


Role of vehicle simulation technologists 

Nothing could be more false. There are scores of experts who have been associated with vehicle 
simulation and teleoperations interface development who have appropriate training and expertise to 
design usable virtual environment displays and have been doing so for years and telling the world about 
their progress in courses on simulation like those periodically offered at MIT and SUNY Binghamton on 
flight simulation. Virtual environments are best viewed as extensions of the technology discussed in 
these courses; in fact, the first head-mounted displays were specifically developed in an attempt to 
replace costly dome-projection flight simulators (Furness, 1986; Barrette, et al., 1990). 

Professional organizations Interested In research agendas 

Another measure of the extent of national interest in the technology are the numbers of workshops and 
conferences sponsored by national professional associations whose members are indeed expert in the 
technologies necessary to make a virtual environment, for example the National Research Council, 
National Science Foundation (Bishop, 1992), the Engineering Foundation (Durlach, Sheridan, & Ellis. 
1991), and NASA (NASA, 1991) and Office of Naval Research (Forthcoming, May 1993). These 
meetings have been and are continuing to be called to help establish national agendas for research. 

When will virtual environments be available? 


Vehicle simulators are available now. 

Virtual environments have been commercially available as flight simulators, for example CAE fiberoptic 
helmet mounted display (Barrette, et al., 1990), for years, but achievement of the required performance 
specifications in practical systems still is very expensive, costing on the order of millions of dollars. Much 
cheaper systems have recently begun to be commercially available. (Division Limited, 1993; W Industries 
Ltd., 1993; FakeSpace; Virtual Research, 1993; Senses Corporation, 1993; Leep Systems, 1993; Virtual 
Reality Group, 1993). The market for the cheaper virtual environment systems has generally tolerated 
much poor performance and manufacturing quality than the flight simulator market. However, poor 
performance and reliability appears to have been partially responsible for the fall of the former market 
leader, the now dissolved VPL Research (Hamit, 1993). 

Cheaper head-mounted systems are sufficient only for video demos 

Most of the extent virtual environment systems using the cheaper, more accessible technology have 
rarely passed beyond the stage of conceptual demonstration to the stage of enabling useful work, 
especially when compared to cheaper existing alternatives. This stasis in a perpetual stage of conceptual 
demonstration and further development leading to further conceptual demonstration is characteristic of 
almost all of the cheaper systems that have been assembled so far. 

The principle reason for this problem is that the technical solutions to the many difficulties in producing a 
personal simulation of sufficient fidelity are still expensive, and many of the research groups investigating 
the technology simply don't have sufficient resources or expertise for adequate development. A second 
major difficulty is that applications of the technology are sometimes fundamentally misconceived. For 
example, the use of a derivative of the DataGlove (Zimmerman, Lanier, Blanchard, Bryson, & Harvil, 
1987), the PowerGlove distributed by Matel ultimately sold only for novelty value and failed to endure as a 
commercial product because its software applications proved physically very tiring to use and were never 
shown to enable uniquely any desirable activity. Unfortunately, exploratory software development by 
outside programmers which might have solved some of the implementation problems was discouraged 
through a variety of technical means by the initial distributor of the DataGlove (Zimmerman, 1992). 

Marketing problems , , 

The difficulty encountered by the PowerGlove project is characteristic of many of the apparently evident 
application areas of virtual environment technology: those advocating and sometimes even developing 
virtual environment displays for a particular application fail to fully understand the performance required of 
both the technology and the operators for successful use. Field use of the viewing technology can be 
especially difficult as illustrated by attempts to use telepresence interfaces in harsh environments such as 
Antarctic (Stoker, 1993). As shown by the experience of the flight simulation community, this 
understanding for a single application environment can require considerable human factors and 
engineering expertise and experience (CarduIIo, 1993), a requirement frequently underestimated by those 
suggesting extensions of personal simulators into other domains. 
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^ro^^advocating^euse of virtual environment displays generally have the significant task of 
demonstrating that such displays can be produced with sufficient symbolic, geometric and dynamic fidelity 
to enable useful work at an accessible price. In fact, as discussed above, much of th ® ^notogy 
embodying virtual environment displays is not new but may be directly traced to developments in vehicle 
simulation dating from the 1920's and teleoperation technology dating from the 1940 s. Consequently, the 
reasons why virtual environments have not become a major commercial product out side of flight 
simulation in the last 30 years is a significant question that must be answered. 

Why have the related applications In telepresence not caught on? 

This question is particularly salient for many telepresence applications which significant^ overlap 
synthetic virtual environment displays based on computer generated scenes with respect to jhe head- 
referenced displays that both use. Such head- referenced displays were first implement at Ph. co in the 
early '60's and extensively advocated for space and other applications in widely circulated journals and 
magazines? for example, Aeronautics and Astronautics (Bradley. S nee the key .nnovanons ot the 

display technology are human interface issues, the reasons for the failure for earlier diffusion irrt0 
numerous possible applications are most likely associated with the cost and performance character^ 
of the human interface Some of the earlier discussions of the limitation on the viewer technology are 
strikingly contemporary yet date from the 1960's. Goertz's discussion about why a 1000 line TV system is 
at least 165 time poorer than the human eye, even disregarding the great difference in available contrast 
ratios, is especially revealing (Goertz, et a!., 1965). 

Technical solutions to the resolution problem. Advances in toom-mounted displays 
(McDowall Bo las, Pieper. Fisher, & Humphries, 1990), improved interfacing techniques and6dofhacker 
characterizations (Adelstein, Johnston, & Ellis, 1992) may provide a solution to the resolution problem as 
well as the transport delay problem that is one of the principle constraints on practical use of virtual 
environment systems. However, examples of practical use of virtual env,ronmen ^ * 

remain isolated for displays in the moderate to low price range, for example less than about $150,000 for 
a complete system These displays potentially can provide a compact format for personal training 
simula^oro oThand- he Id systems such as Hand Held Maneuvering Units for use in space (Brody. Jacoby. 

& Ellis 1992) or Stinger anti-aircraft missile launchers (Jense, & Kuijper, 1993), but even 
applications are still essentially conceptual demonstrations awaiting further improvements in the 
inexpensive virtual environment systems. 

Demonstration of real utility: comparison to panel mounted formats. A key missing element 
in many ofthe applications areas is a rigorous comparison of user performance with a virtual environment 
display^ contrasted with performance achieved with a well-designed, possibly stereoscopic Panel runted 
substitute. Such panel mounted alternative hardware formats are publicly viewable, available L™th h gh 
resolution and currently generally cheaper than virtual environment systems. When su ® h ® 0I ^P a ^ t ‘'l® 
studies are suqqested, VE developers often complain that their systems are not yet ready for such testing. 
There is clearly truth in this claim as most of the head-mounted visual displays systems cannot meet sue 
basic specifications such as the recommended number of scan lines per character of displayed text 
(Weintraub, & Easing, 1992). But unless such comparisons of alternative display format are made, the 
Dotential benefits of the new technology will never be known and the users and supporters of the 
development will have to wait indefinitely to learn whether the promised wonders will even practica y 

materialize. 

Some commercial applications. Never the less, some apparently economirally ^ccessfuf 
applications have appeared. In Japan Matsushita Electric Works in Osaka has used the VPL Eye Pho e 
svstem as a successful marketing tool to help sell custom-designed kitchens and cabinetry. This 
arol?catfo S n1s S a^ex^plTof the Architecture walk-through" demonstrated by Prof. Brooks' group at UNC 
(Airey Rohlf! & Brook S P 1990). Also "virtual reality" video games have been distributed by a British 
Lmnanv railed W Industries under the name of Virtuality and may be commercially successful. But 
commercial success of companies working in this field is ^rtainly not guarai^ed due to ^p.dly changing 

technical factors, i.e., the availability of better display technologies, and [J® the manu1a 9 rturers in the 
manufacturer i.e., Sony (Annonymous, 1993), might enter the market Most of the ^/adurers in tne 
U S are small startups y and VPL, once acknowledged as the industry leader, has essentially gone 
hankmnt duTto overextension (Hamit, 1993). The ease with which a developer may lose focus when 
woSg in ms area may be a characteristic ol the technology itself. Being a common, canons medium 
virtual environments appear to be useful for practically everything. 
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Sources of technological strength. This apparent strength is in reality a significant weakness. 
Technologies derive their strength not from their generality but from their uniqueness. That which makes 
them truly useful is that which makes them distinct(Basal!a, 1988). Aircraft simulators are not useful 
because they can simulate a generic aircraft, but because they can simulate a Boeing 747SP. However, 
as mentioned earlier, such specific simulation is achieved only after considerable engineering 
development and human factors tuning and testing. As similar efforts are brought to other potential 
application areas, virtual environment displays will move from the demo room to the desk-top. cost 
reductions will accompany enlarged markets and the number of economically viable applications will grow 
as compact, personal simulators are customized to solve specific tasks. As major corporations enter the 
head-mounted display market and promise to radically lower the cost of a display (Annonymous, 1993), a 
variety of new applications may be explored. 

It must however, be said, the VE industry has not yet found its "VISICALC" - the "spreadsheet" 
application whose invention created the microcomputer industry because thousands of potential users 
recognized in it an accessible, new, affordable tool that enabled them to do their existing jobs better and 
to imagine solutions to previously intractable problems. 


Findinq a "Visicalc-like" application which would underscore obvious benefits from virtual environment 
displays is especially important because their use also brings risks and costs. Like the flight simulators 
which were their predecessors, extended time in virtual environments can produce nausea and altered 
visual and visuomotor coordination as lasting aftereffects which can interfere with automobile driving and 
other aspects of normal life in the physical environment to which all users must ultimately return. 
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Chapter 2: State of Knowledge, State of Technology, 
Limitations, Research Needs, and Implications for 

System Requirements 


A Virtual Environment (VE) is an interface media. Therefore, human performace is one of the important 
considerations in defining requirements for a VE. In this section, we consider visual, auditory, haptic and 
vestibular perception. Briefly the state of research knowledge relevant to VE is presented. Research to 
provide the knowledge needed to more fully define the requirements for a VE system are outlined. The 
implications the state of research knowledge has for VE systems requirements are specified. 

Chapter Organization 

Vision and Visual Perception 4U _ .. 

Visual Image: Luminance, Contour, and Color on the Retina 
Visual Scene: Segregated Interpretation of Images 
Visual World: Spatial Interpretations of Scenes 
Head-Mounted/Head-Referenced Displays 
Spatial Perception and Orientation 
Visual-Vestibular Research and Motion Sickness 
Haptic Interfaces 

Haptic Perception 

Haptic Interface Design and Control 
Gesture Recognition 

Audition and Virtual Acoustic Displays 

Improving Human Performance 
Issues for Audio Communications 
Spatial Orientation and Situational Awareness 
Synthesizing Non-Spatial Auditory Icons 
Issues for Hardware/Software Development 
Overall System Issues 
Host Computers 
Networks 

Models, Imagery, and Other Data 
Software Tools 
Domain Analysis and Design 
Spatial Position/Orientation Trackers 
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Vision and Visual Perception 


Background . 

Vision dominates performance and perception in a VE. This key role has 3 aspects: characteristics of the 
visual image, structure of the visual scene, and visual consequences of interaction with the scene. 


Visual Image: Luminance, Contour, and 
Color On The Retina 


State of Knowledge / Technology 

Visual image properties are well understood from previous applications in flight simulation and the design 
of optical displays inlcuding HUDs, photographic or video displays. Flight simulation displays have higher 
information bandwidth than those currently used in VE. Significantly, known visibility requirements for text 
on HUD displays are not satisfied by most VE helmet displays. The specific trade-offs between the purely 
visual parameters that could be made for improved design of VE displays are not in general known or 
demonstrated for target tasks. For example, because of the binocular overlap between the left and right 
eye images need not be complete, monocular fields exceeding 60° may only rarely required. 

Research Needs , ___ „ _ 

Display resolution: Precision visual tasks will require better image properties of small CRT display 
systems providing daytime luminance levels. Visual performance needs to be assessed with parameters 
likely to be provided by future display systems which may use nonstandard pixel layouts, variable field 
resolution, and field magnification to optimize allocation of computer graphics processing. For example, 
the benefits of inserting higher resolution imagery into the central visual field need to be explored. 

Stereoscopic properties, for example characteristics of display separation, need to be studied to measure 
likely visual fatigue with protracted used. Trade offs between the binocular overlap between the left and 
right eye images and total field of view need investigation. 

Trade off studies between visual image parameters, field of view, resolution, stereo disparity, color gamut, 
need to be conducted in task environments since in existing systems provision of full fidelity of all 
parameters is impossible. Trade off studies using "boom" mounted displays -- which can provide much 
higher resolution, but are much harder to move naturally -- are needed to establish specific visual 
requirements for NASA tasks and measure depth and direction sensitivity, discrimlnability and bias, as 
well as target recognition and detection. 

Research should be conducted to examine the use of synesthetic simuli that, for example, transform a 
visual signal such as the brightness into an auditory or tactile signal. VE provides enormous flexibility for 
such studies and should be exploited to develop new communication channels for sensory stimuli. 


VE System Requirements 
Resolution: 


Field of view: 
Magnification factor: 

Binocular overlap: 
Disparity range: 


2-3'/pixel in central 20° field; Many manufacturers specify display resolution 
of their devices, but practical testing has indicated in the Ames lab that the 
displays do not meet the published specifications, Standardized testing is 
needed. 

Total field of view >60°. 

1 , but zoom lens compensation for poor resolution is needed, Visual 
disturbances due to zoom need to be determined. 

20-30°, trade off of binocular overfap;total field of view needs further study, 
unknown; magnification aggravates fatigue; countermeasures need study. 
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Visual Scene: Segregated Interpretation of Images 


Visual images are perceived to be segregated into subparts based on image processing rules that are not 
completely understood, but which collect visual attributes such as color and shape and segregate the 
image into regions. This segregation is accomplished by both simultaneous and seria information 
processing. VE displays must have sufficient fidelity to allow this processing to take place. Most existing 
displays do not provide sufficient visual fidelity for these processes to take place normally. 


Rososrch Needs 

Since VE will only be able to present somewhat degraded low level visual cues such as contrast and 
stereopsis, the capacity for viewers break camouflage - to segregate foreground from background - is 
likely to be less than that with natural images from real environments. Accordingly, visual segregation 
with degraded image quality and dynamics should be studied and enhancements to overcome difficulties 
should be developed. This is an area where knowledge of texture perception can be used to help control 
the visual clutter in images. 

Specific trade offs of image properties may be studied to improve image segregation, for example color- 
coding can be used to substitute for poor stereoscopic characteristics which normally reveal concealed 

objects. 

Studies of use of other sensory modalities to assist image segregation should be conducted so as to 
improve the VE user's sense of the spatial and control context in which they find themselves. Knowledge 
of context can improve the users’ ability to segregate a noisey or imperfect visu al image. 


VE System Requirements 

Visual resolution needs to be increased to levels approaching that of a standard desktop CRT such as a 
Macintoch monitor so a wide variety of textures can be presented. 


Visual textures, the integrated spatial distribution of contour and shading across the visual image, needs 
to be quantified and computational models are needed to predict their visibility in VE displays systems. 


Visual clutter: visual textures that can assist image segregation in VE displays which fail to meet known 
visual performance specifications for other tasks need to be identified. 


Rgfgro nces 

Treisman, A. (1985) Preattentive processing in vision. 
Processing, 31, 156-177. 
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Visual World: Spatial Interpretations of Scenes 

SMrea at ed visual scenes are normally interpreted as an external world populated by actors and objects 
located in space. The spatial interpretation of visual images is highly dependent upon the moving 
properties of the image, in particular those motions that are consequences of the observer himself. The 
patterns of image motion that are associated with observers' movements provide much of the necessary 
information for guidance through a cluttered environment and have provided the basis identifying the visual 
cues to motion, space and causality. In this field, researchers have investigated the natural linkages 
established between properties of image, or object motion, object position and orientation and complex 
normal behaviors such a walking, perception of self-motion, object avoidance or manipulative interaction. 
Computation models of the spatially related behaviors in the visual world have been developed to assist the 
visual design of vehicle simulators. 

Visual information is not only important for local navigation and perception of self-motion while traversing 
an environment but also for global path planning and route selection. Visual orientation is also important 
tor more integrated tasks in which subjects use visual aids such as maps to maintain their internal 
representation of the surrounding space and assist planning of future activities. 


R6S63rch ^4eeds 

Models used to predict phenomena in the visual world, such as vection, illusory self-motion due purely to 
visual stimulation, need to be applied to the VE. Most previous work is relevant to simulators that are 
entered rather than worn. Experimental studies checking model predictions are needed. 

Subjective and objective operator reactions to approximated kinematic and dynamic 
environments. How far can a simulation deviate from correct physical modeling and still appear to be 

realistic? 

Visual stimuli that induce apparent self-motion should be evaluated for integration into VE displays 
designed to create apparent movement through virtual spaces. 

Since visual textures are a major cue to purely visual spatial perception, the abiity of users to make purely 
visual texture based spatial judgements should be investigated and enhanced by necessary image 
processing to overcome deficiencies is visual displays. 

Since imperfect and slow dynamics of VEs can lead to significant difficulties for users to main their spatial 
orientation within a simulated larger environment, sensitivity studies of visual disorientation are needed 
Orientation aids to compensate for these difficulties should be developed to allow developers to simulate 
highly detailed real environments when such detailed simulation is required. These aids should assist 
users switch between ego and exocentric frames of reference which will be needed for efficient 
interpretation and control of objects in the simulated environment. 


R!STta R s5Slo"be identified to determine if the visual world is presented with adequate fidelity so 
as to allow the viewer to perceive causal interactions between actors and objects. 

These standardized tasks are similar to those used to obtain FAA qualification for a flight simulator, but 
have not been identified for VE applications in other areas. 
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Head-Mounted / Head-Referenced Displays 


State of Technology 

A VE display is a visually dominated device. Though other sensory modalities may contribute essential 
information the illusion of being at another location is primarily visual. VE visual displays have primarily 
been head-mounted stereoscopic devices which only can present a visual image with sufficient field of view, 
resolution, color gamut, luminance and dynamic fidelity with great difficulty and expense, e.g., the CAE 
Fiberoptic helmet mounted display (FOHMD). While helmet mounted displays are inherently somewhat 
obtrusive they can be more flexible and less expensive than alternative back-projection systems e.g., the 
"CAVE" of University of Illinois. In fact, the FOHMD was designed as a cost-effective alternative to a dome 
projection flight simulator. Experience with existing VE helmets as well as current ergonomic and human 
factors knowledge all support the fact that VE head-mounted display technology in general fails to meet 
minimum requirements of image quality, weight, and packaging to allow VE operators to use the displays for 
long periods of useful work. 

Alternative head-referenced displays have been developed to solve the weight and resolution problems, 
originally by Ray Goertz but more recently by FakeSpace Labs. These displays involve a using boom 
mounted CRT to present the visual image generated for the VE.Since the boom can carry the CRT and 
optics weight, much improved imagery can be provided by higher resolution monitors and optics that in 
other formats would be too head to use. These types of displays may either be actively controlled like a 
submarine's periscope or may be made to passively track the head of the user. Though in either case the 
resulting display system is more cumbersome than a head-mounted one, it can deliver the required visual 
performance for useful work. The added inertia and mounting linkage, however, interferes with the illusion 
of being directly present in a VE. 

Research Needs 

Systematic comparison of panel-mounted, head referenced and head mounted formats need to be made 
to determine which applications require which format. Those that can usefully adopt head-referenced 
boom displays are those for near term products. 

Advances in miniaturization of high definition video displays needs to be encouraged since the lack of a 
small (1") high-definition monitor with greater than 1000 X 1000 line resolution complicates the 
presentation of a visual image with needed resolution. 

Alternative optical techniques for providing high resolution imagery should be investigated, i.e. fiber 
optics, relay lenses, video mixing systems. 

Techniques for correcting for the inertia and cumbersome linkages in boom mounts for head-referenced 
displays should be investigated. Minimum kinematic fidelity requirements for display mounts should be 
established. 

High resolution electronic image and video standards need to be established to provide a technological 
infrastructure that will encourage the development of display technology utilized by VE. 


VE System Requirements 
(See visual display requirements) 
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Spatial Perception and Orientation 


Background 

Coordinated presentation of stimuli in different sensory modalities enhances the illusion that an observer ^ 
is in a virtual space. There are 4 factors influencing this illusion: 1) The match of synthetic sensory stimuli 
to natural stimuli, 2) The correlation of changes in all spatially dependent stimuli, 3) The naturalness and 
completeness of motor interactions, and 4) the normal correlation of movement and its sensory 
consequences. The completeness of the illusion can measure overall VE simulation fidelity. 

State of Knowledge iL 

Spatially related physiological reflexes in VE are well described by standard biomedical techniques. 

Some reflexes, such as, the vestibular-ocular reflex, postural balance reflexes, and the looming reflex, 
may be used to provide objective measures of the extent to which inhabitants of a VE behave as if they 
were in a real environment. More complex psychological responses to being present in a space also can 
measure the immersion illusion, i.e. apparent location of the horizon in a variety of reference frames, the 
spontaneous avoidance of "visual cliffs" or the presence of environmentally induced circular and linear 
vection (self-motion). The correlated sensory stimulation associated with natural movement combine 
synergistically to improve the sense of spatial immersion and provoke the above spatial responses 

Inhabitants of physical space are generally sufficiently well adapted so that spatial tasks may be 
successfully accomplished. For example, the egocentric direction of target may be determined so that 
they may be manipulated or moved toward in a coordinated manner. Subjective measures of direction 
and depth, that have been studied in physical environments, may be adapted for studies in VEs to 
measure their spatial fidelity. Static measures include accuracy of perceived egocentric direction and 
apparent depth. Dynamic measures include 3D tracking, pick and place tasks, and reports of perceived 
of egocentric and exocentric motion. 

The essential character of the immersing spatial illusion provided by a VE is its interactivity. Similar 
interactivity, present with usual human-computer interfaces, has been the object of many "useability 
studies. These provide models for parallel issues in VE and 3D generalizations of 2D graphics interfaces. 

Research Needs 

Spatially related physiological and psychological responses need to be validated as measures of 
simulation fidelity in VE 

Engineering demonstration environments should be developed to allow formal comparison of 
performance with VE interfaces to spatial data with well-designed panel-mounted alternatives. Existing 
panel-mounted displays may provide faster and cheaper interfaces for some spatial tasks. Exactly which 
types of spatial tasks can uniquely benefit from the VE interface need to be determined. This 
determination can be accomplished by analysis of task-specific, spatially oriented behavior in the field and 
laboratory analysis of how the target task may be more efficiently accomplished with VE technology. 

Applied tasks such as aspects of satellite servicing with a hand-held orbital maneuvering system need to 
be developed to determine if VE can provide a useful simulation and training capacity for the task in 
question. This type of performance measure is similar to the acceptance testing done for flight simulators. 

Svnerqistic combinations of stimuli, such as vision and correlated sound, need to be used to improve the 
overall spatial response to the stimuli which would otherwise be incomplete due to known imperfections 
into the VE display technology. Studies should be conducted to investigate the potential of substituting 
information from one sense for that in another, such as visual or auditory presentation of force. 

VE System Requirements , , . . . . . 

Simulation fidelity requirements for VE applications depend upon the specific task environment but shoura 
all the VE user to be spatially oriented at least as well as in the corresponding real environment, but if he 
VE user is not better oriented in the VE than he could be in the real environment, the VE systems would 
have failded to provide its potential benefit. Existing simulation fidelity requirements for flight simulators 
may provide initial guesses of what VE requirements are for personal simulators that are worn. 

O prp ness 
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Visual-Vestibular Research and Motion Sickness 


Human locomotion and movement involve a dynamic sensory-motor adaption to the background gravity 
on Earth. Disruptions of the normal patterns of sensory feedback from vision, vestibular sense, touch, 
somatosentosensation, proprioception combined with motor corollary during locomotion and other 
coordinated movements. In particular visual-vestibular conflict can arise from several sources and can 
cause serious performance degradation as well as motion sickness. 

State of Knowledge 

Visual motion displays can cause malaise similar to simulator sickness where visually-induced vection 
contradicts the vestibular system which signals that the head/body is not moving. Conversely, feedback 
delays or inaccurate signals from head trackers cause the display to be updated inaccuratly so that 
vestibular system signals self-motion while the visual system does not. Distortions, low-resolution, and 
small fields-of- view in the visual display can also create visual-vestibular conflicts. The ability to generate 
arbitrary vehicle motions in VE may also cause unanticipated sickness. Adaptation to VE can cause post- 
VE malaise, sickness or even perceptual errors with potentially disastrous safety consequences. 

Serious performance errors often occur in present state of the art VEs. Spatial calibration is generally 
empirical as the many sources of error are not well quantified (eg. simulated eye separation, display 
distortions) that can cause reaching errors and other sensorimotor dysfunction. Sensorimotor control is 
difficult and error-prone in systems with low resolution and long temporal delays. 

Research Needs 

Future VE development will require predictive models of human motion sickness. For example, given a 
particular combination of visual, vestibular, and other stimuli, what is the probability of getting sick? After 
what length of time? What types of visual and vestibular stimuli cause motion sickness? In particular, what 
types of visual-vestibular conflicts are tolerable and which are not (e.g., minimum head position update 
rate)? These models should also provide a better understanding of what types of stimuli could be 
provided to counteract motion sickness (e.g., tread mill or other devices to simulate actual motion stimuli). 

Countermeasures to motion sickjness in VE should be studied using existing drug treatments but 
extending to specifically designed training environment which could be presented as VE. VE should be 
explored as an adaptation environment for exisiting forms of motion, simulation, and space sickness. 

Effective VE design will also require predictive models of human performance in 3D perceptual tasks that 
provide an understanding of the minimum visual-vestibular information necessary for accurate self- 
motion perception and the maximum adequate for asymptotic performance. For example, how do visual, 
vestibular, and other stimuli combine to generate accurate performance? What visual parameters are 
necessary for accurate performance? Will an additional minimal vestibular stimulus enhance performance 
over visual stimuli alone? What is the minimum amount of information necessary to generate asymptotic 
performance? What are the information-performance trade-offs? What information is unecessary so that 
computation or data storage resources are not wasted? In addition to enhanced visual display and head 
tracking methodologies, cleaver use of vestibular and other stimuli may be fruitful and need evaluation. 

VE System Requirements 

High spatial and temporal resolution visual display with accurate and fast head tracking, are required. 
Adequate visual spatial and temporal resolution will be necessary to minimize visual-vestibular conflict. 
The exact parametric values are at present unknown and will require empirical measurement and model 
development. In addition, effective countermeasures within the VE design will be necessary to avoid or 
eliminate either sickness either before or after VE work. VE should encorporate hardware to aid normal 
visual-vestibular correlations during movement such as, motion platforms and treadmills. 
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Haptic Perception 


Human haptic perception requires the active integration of information on touch from the surface of the 
skin as sensed by tactile receptors with information on limb displacements, velocities, and forces as 
detected by joint, muscle, and tendon receptors. When integrated together, stimuli to these various sense 
organs contribute to the overall “feel" of the material properties (e.g., shape, hardness, weight, etc.) of real 
objects when we physically interact with a real environment. A haptic VE strives to emulate these real 
mechanical characteristics by stimulating these sense organs. Applications of interest for NASA include 
telemanipulation and force reflection for fly-by-wire aircraft. 


State of Knowledge . x . , u . 

Haptic interlace research at laboratories in the US and other countries has focused separately either on 
the tactile sense alone through stimulation of the skin, or on the whole-limb displacement, velocity, and 
force sense of mechanical dynamics when coupled to electromechanically powered joystick and 
exoskeleton devices. The physiology of both cutaneous and musculoskeletal haptic sensory organs and 
the psychophysics of tactile perception have received considerable research attention over the last 
several decades. By comparison, until very recently, human perception of dynamic mechanical 
characteristics has seen relatively little work beyond basic studies of kinesthesis (perception of limb 
displacement) and static force discrimination. Several research groups are beginning to investigate 
temporal and spatial psychophysical discriminants of simple impedances (i.e., springs, dampers, inertias), 
forces, and displacements [1,2, 3]. Among their goals is to contribute to quantitative design guidelines for 
future haptic interfaces. 


The most pressing research problems are in the areas that will lead to quantitative performance 
specifications and combined human-system models for haptic interfaces. Research into the . 
psychophysics of haptic perception must be developed to include the aspects relevant to spatial and 
temporal whole-limb sensation of mechanical properties beyond those derived purely for the sense of 
touch. A necessary first step is research on the perceptual dynamic range and resolution requirements 
for simple mechanical quantities such as displacements, velocities, forces, and impedances. 

A related issue is how the intrinsic mechanical properties of the haptic interface hardware alter human 
perception of those basic quantities. Similarly, basic computer control parameters such as update rate 
and temporal latencies that arise in modulating haptic interface mechanical output (parallel to CRT refresh 
and persistence) must be examined. 

Research that can be carried out at NASA encompasses basic psychophysical experimentation that 
depends on high performance haptic research apparatus as well as psychophysical and related human 
performance studies involving multisensory interfaces to augment haptic display capabilities. 


VE System Requirements . 4 ... 

An understanding of the type and detail of dynamic mechanical information from the environment that 
humans can integrate will lead to appropriate design specifications for general and application-specific 
haptic systems as well as individual actuator and sensor components. Psychophysical research will also 
lead to quantitative models of human haptic perception that are vital to effective control algorithm design 
and analysis (see section on Haptic Interface Design and Control). 

Quantitative design specifications are needed to guide technological goals for future haptic interfaces— 
both to drive the market to achieve these goals and to ensure that effort is not expended to unnecessarily 
exceed these specifications. 


1. Fasse E.D. (1992). On the Use and Representation of Sensory Information of the Arm by Robots 

and Humans. Ph.D. dissertation, Department of Mechanical Engineering, M.I.T., Cambridge MA. 

2. Jones, L.A., and Hunter, I.W. (1992). Human operator perception of mechanical vanables and their 

effects on tracking performance. In H. Kazerooni (Ed.), Advances in Robotics (pp. 49-53). New 
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Tan H.Z., Pang, X.D., and Durlach, N.l. (1992). Manual resolution of length, force, and compliance. 
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Haptic Interface Design and Control 


Unlike v?sual or auditory VE displays, haptic interface hardware typically transfers information to the same 
body part that delivers the human operator’s (e.g., manual) response. Consequently, a distinguishing 
attribute of the haptic display is that the transfer of information back-and-forth between the human _ 
operator and the VE entails mechanical power exchange through the interface and attached human s limb 
seaments [11. The implication is that the modulation of the mechanical characten sties of the haptic _ 
interface by computer control and the mechanical characteristics of the human limb under the human s 
control will alter mechanical power flow and hence affect information transfer. Thus, the development of 
effective haptic displays for VEs will depend on a thorough understanding of the effects of interface and 
limb mechanical impedance on human sensory input capabilities. 

State of Knowledge / Research Needs 

One area requiring research is control algorithms for haptic displays. Much of the control research has 
benefited from recent analytic tools for robust stable telemanipulator control [2], Research into the 
requirements for stable interface behavior is one prerequisite for acceptable haptic presentation quality. 
Thus far, though, this work has not been applied for the purpose of human sensory interaction at 
frequencies beyond the bandwidth of manual output. 

A second line of research must be directed to the development of hardware subcomponents with 
appropriate performance capabilities for use in haptic interface systems. Several academic and 
commercial groups have concentrated on the construction of general purpose and application-specific 
haptic interface hardware. Due in part to the lack of comprehensive objective design specifications to 
help develop the market, a shortcoming encountered in many of these devices has been the lack of 
suitable, commercially-avaitable actuators and sensors on which to base their designs. The most 
successful future designs will benefit from research into radically different sensor and actuator concepts 
that will be geared specifically toward haptic interface applications. 


VE System Requirements , . ,. .. 

General performance specifications for haptic VE applications are listed below. The type of application— 
whether the intended use is dominated by purely tactile or whole-limb kinesthetic and force information 
display — should be the major consideration in developing quantitative specifications for haptic displays, it 
is important to note that, especially for the limb sense of dynamics, little experimental data is available at 
this stage to develop these specifications. 


Motion resolution: 


Motion range: 
Force range: 


Frequency range: 


Impedance range: 


Minimum relative displacements for output to human is on the scale of 1 pm 
for tactile perception and 0.1 mm and greater for purely kinesthetic 
applications. 

Application specific. Depends on which body segments are coupled to the 
interface. 

Application specific. The interface may need to be strong enough to resist an 
human motion: -1000 Newton (225 lb) maximum for the arm at 0 Hz and 
trailing off to -25 Newton between 3-10 Hz when the arm is braced in a 
stationary posture. 

Not fully known — the subject of psychophysical research. The interface must 
track limb output with negligible lag, implying a minimum analog bandwidth of 
50 Hz and computer control update rates of at least 1 KHz. Muscle velocity 
receptors are sensitive to vibrations of 200+ Hz; skin receptors to frequencies 
as high as -5 KHz. 

Equivalent to resistance to motion. Unknown — requirements need to be 
determined from extensive psychophysical research. Ideally— though not 
practically — impedance should span from zero to infinity. Zero impedance 
would make the interface feel completely “invisible", while infinite impedance 
would make an interface feel absolutely rigid. 


I^Kazercxfni, H. (1992). Human -robot interaction via the transfer of power and information signals. IEEE 
Trans. ’ Systems, Man, and Cybernetics, SMC-20, 1992,450-463. . 

2. Colgate, J.E, Grating, P.E., and Stanley, M.C. (1993, October). Implementation of stiff virtual wails in 
force reflecting interfaces. Submitted to IEEE-VRAIS, Seattle WA., Oct. 1993. 
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Gesture Recognition 

State of Knowledge/Technology includes locomotion and manipulation, as well as 

Gesture is purposeful bodily c jdeas emotions etc or convey a state of mind or intention, 

movement which can expressor^phasize id , morons, y essentia , t0 user -computer 

*5® SSSS.'SSSti 

on the moon all require gesture. 

The ability of a system to recognize Slh^sykem via geSu°re S ' Wrth VEs P fhe° 

paces, determines the ability of the user to t d on8 but recognition of such gestures is 

gestures used in the real world can equally apply 'naw m put®done butreo^nno ^ hag ^ 

currently very limited. The most Mam^ gestures in VEs to 
used to control the computed view to be p hand shapes have been non-intuitively mapped to 

date have, however been primitive of the most usefu i 

commands such as "hide menu Currently - voice ns thougni oy ^me ^ ^ ^ bg used tQ jssu8 

command gestures in VE since JJ®^®* ?! f h a keyboard 1 menus or buttons. Natural gestures, those 
complex verbal commands without the use of a keyboa , enu ^ ^ re { t0 their pote ntial as 

used in the real world, have not yet been studied to a y g gesture interactions have been limited 

elements of communication with C0 ™P^®^ up massless virtual objects. 

to pointing in directions or at ^^^thnunh one insoi^VE encourages users to touch the hearts of 
and pointing and firing weapons (a though i one inspired vte « trackers, or can be done on 

angels!). Walking in VEs Is currently tamted to a radius interactions with real 

in virtual ones, and exploitation ol virtual gesture Interact, ons has 

not yet occurred. 

Hurnan perfc^ance research is needed to provide an integrated theoretical iramework for the use of 
gesture to interact with VE systems. 

Given that VEs can potentially utilize all possible gestures, human performance research is needed to 
derive maps of "VE gesture space" for real world applications. 

Joint human performance and computer science research Is needed to develop gesture recognition 
systems capable of recognizing natural, dynamic gestures of all kinds. 

vl systEmt* gesture recognizers must not impose encoded gestures upon users. 

Gestures recognized by VE systems must be derived from the targe, user domains, and intuitively 
understood by user domain experts. 

Gestures learned on other systems. 

X»e^ data^must to lactored^into VE gesture 

interface designs. 
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Virtual Acoustic Displays: Improving Human Performance 

The synthesis technique used in spatial auditory displays involves the digital generation of stimuli using 
Head-Related Transfer Functions (HRTFs) measured in the ear canals of individual subjects in an 
anechoic (non-reverberant) environment. In most current systems, from one to four moving or static 
sources can be simulated (with varying degrees of fidelity) by filtering incoming signals with HRTF-based 
digital filters chosen according to the output of a head-tracking device. Motion trajectories and static 
locations at greater resolutions than the empirical data are generally simulated either by switching, or 
more preferably, by interpolating between the measured HRTFs. In some systems, a crude distance cue 
can be provided via real-time scaling of amplitude. 

While current spatial auditory displays have proven benefits, they have only been partially perceptually- 
validated in terms of localization accuracy and realism. Further, their potential can be significantly 
enhanced by advances in hardware and software design driven by evaluations of human perceptual 
performance. 

Status of Knowledge , . 

To date, several primary areas have been identified for improving human performance. These include. 
(I)simulation of acoustical environmental cues to enhance distance perception and reduce perceptual 
errors like front-back confusions; (2) simulation of acoustical cues induced by head and source- 
movements (e.g., via tracking devices) (3) understanding individual differences in HRTFs and 
identification of perceptually-relevant methods for modeling HRTFs to overcome such effects; (4) the use 
of feedback/training and super localization techniques to examine adaptation to auditory displays over 
time Recently NASA-funded studies have demonstrated the perceptual consequences of individual 
differences, the potential of using modeled HRTFs in place of measured HRTFs, and the usefulness of 
reverberation simulation to mitigate externalization errors. Less work has been done in the areas ot 
training/adaptation and the potential benefits of dynamic motion cues. 

Ros63rch Needs 

Critical features of reverberant environments (synthesis of early and late reverberation) must be 
parameterized in order to develop more powerful interactive, real-time hardware / software systems. For 
example the number of early reflections that must be simulated to ensure realistic localization is ^ 
unknown. Also, since the acoustic image gets larger (and therefore harder to localize) with increasingly 
reverberant environments, the trade-off between localization accuracy and realism must be determined. 

In dealing with individual differences, several approaches are possible, including use of non-personatized 
HRTFs from people known to be good localizers, statistically-based models of HRTFs using techniques 
like principal components analyses, and structural models of HRTFs based on the physical shape of the 
ear/body structures. Such techniques can potentially drastically reduce the computational overhead 
during spatial synthesis. However, perceptual validation studies are necessary to ensure that localization 
accuracy remains acceptable in all of these cases before implementation in a hardware system. 

Simulation of dynamic cues from both head and source-motion implies special problems forhar^areand 
software development, including determining required update rates and methods for interpolating between 
measured or modeled HRTFs; these need to be evaluated in perceptual studies to determine the optimal 
methods and hardware specifications. 

Finally, most current studies use relatively naive subjects and may be underestimating u ser s’ ability to 
localize virtual sources. Studies are needed to examine the localization ability of trained sub J e ^s Sl ™r® 
users of VE systems are likely to be highly-trained in many applications. Super localization or the artificial 
enhancement of localization cues that is made possible by a virtual system may also be of use. 


VE System Requirements 

Reverberation synthesis: 

Individual differences: 

Dynamic cue simulation: 

Training & super 
localization: 


Determination of perceptually-relevant cues. 

Design HRTFs useful for the general population. 

Develop and evaluate interpolation and sound movement algorithms. 

Measure how human performance with virtual acoustic display 
improves with training and/or additional cues to enhance localization. 
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Virtual Acoustic Displays: Issues for Audio Communications 


A 3-D auditory display can provide a listener with the sound source separation that occurs in normal, 
binaural hearing. Radio communication personnel (e.g., those in shuttle launch operations) must 
frequently monitor more than one communication stream simultaneously, over a single earpiece. This 
combination of signals at one ear leads to difficulties in segregating sources, and requires a desired signal 
to be transmitted louder than would be necessary with two ears for an equivalent level of intelligibility. 

This is also true for a single communication source heard against background noise. Louder levels result 
in increased fatigue during long-term exposure, thereby diminishing safety. 

The 3-D auditory display separates input radio communication streams as if they were actual sound 
sources heard in a real room from different places. The importance of harnessing this binaural advantage 
has been recognized in, e.g., hearing aid development, but practical implementation within radio 
communication systems has been lacking. 

State of Knowledge /Technology . 

Methods for implementing 3-D sound displays using computer platforms such as the IBM-PC have been 
under development since the 1980s. However, these systems are too cumbersome for most aeronautical 
and space applications. Recently, a self-contained device with non-volatile, interchangeable memory was 
developed at Ames for use at Kennedy Space Center that spatializes four sources. An intelligibility gain 
of 6-7 dB was measured for identification of call signs against a background of multiple speaking voices. 


Research Needs 

While studies have been conducted that evaluate the optimal position of one signal vs. noise, additional 
studies aimed at understanding perceptual segregation are needed to determine the optimal positioning of 
multiple sources, and the relationship between number of sources and intelligibility. Another need 
involves hardware/firmware requirements. Some progress has been made in simplifying spatial cues so 
that smaller, faster firmware/hardware can be developed. However, additional miniaturization (from 19 x 
22" x 2") is desirable for some applications. Hardware development for concurrent radio/telephone 
interfaces is also desirable. Perceptual studies on the relationship between available frequency response 
from the particular communication system used and the effectiveness of the technique are necessary. 
Finally, determining the optimal set of headphones/headsets for a given application Is a fundamental 
requirement. This could include combining this technology with active noise reduction techniques. 


VE System Requirements . . , 

HRTFs: A limited set of HRTFs usable for the intended user or group, optimized 

around (1) usable communication frequency response, and (2) 
positions most useful for increasing intelligibility. While only a limited 
set of HRTFs are needed, obviating the need for computer interface, 
these should be interchangeable rather than hard-wired. 

Communication system Isolated (low-cross talk) sources must be obtained from the interface, 
interface: with consideration of incoming frequency response. “Fail-safe" design 

necessary, for operational use. User ability to control levels, 
distribution of sources, and talk-back. When necessary, ability to take 
radio, telephone, warning system, active noise cancellation, and 
situational awareness audio simultaneously. 

Headset: Stereo headsets that are acceptable to communication personnel, and 

that allow simultaneous monitoring of surrounding ambient 
conversations, while (possibly) equipped with active noise cancellation 
technology. 
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Virtual Acoustic Displays: Spatial Orientation and Situational 

Awareness 


The auditory system as an alternative or supplementary information channel for situational awareness 
the identification and localization of external objects -- has great potential in both virtual reality and head- 
up applications. In contexts where considerable demand is made on the visual modality, spatial hearing 
via 3-D sound displays allows either additional channels of information to be conveyed, or acts as a 
positive "redundant" source of information to a visual display to reduce the potential of error. 

State of Knowledge / Technology . _ _ . . , ,. 

Methods for measuring the head related transfer functions (HRTFs) necessary for 3-D sound simulation 
have been under development since the 1970s. Measurement techniques are time consuming and 
usually require anechoic environments. Data on perceptual performance related to the use of 3-D sound 
tor situational awareness has only begun since the mid-1980s. This research has shown that judgments 
of the location of virtual sound sources are reasonably close to judgments made in normal hearing, except 
for greater magnitude of reversals and distance errors (see "needs", below). The usefulness of these 
cues in current situational awareness implementations have been already demonstrated at Ames, e.g.,3 
3-D auditory TCAS display significantly reduced head-up target acquisition time in a flight simulator study. 


Ros^srch Needs 

Current results show a higher magnitude of perceptual errors with simulated 3D sound, compared to real- 
world spatial hearing. This includes image reversals (front-back or back-front) and distance errors, 
including hearing sound within the head. Other perceptual errors sometimes show up in the form of overty 
high azimuth misestimation for certain positions, and inability to process vertical cues. These needs are 
being currently addressed by research into (1) a more complete representation of the acoustic 
environment via reverberation cues, (2) the use of head-tracker interfaces for simulating head-movement, 
(3) HRTFs that are perceptually validated for a large proportion of the general population, and (4) methods 
for quickly measuring "custom” HRTFs. 


Another need involves hardware/firmware requirements. Some progress has been made in simplifying 
cues so that smaller, faster firmware/hardware can be developed. Two recent studies suggest that 
simplified HRTFs are as perceptually salient as actual measurements. However, an important need is to 
derive, then simplify, the methods for determining essential localization cues that would be useful for t e 
general population, or for a particular individual. 


VE System Requirements 

Perceptual validation: 


A set of HRTFs usable for the intended user or group; knowledge of the 
potential localization error with the intended sound source. This 
includes knowledge of azimuth/.elevation error, reversals, etc. 


Head tracking, 
interpolated HRTFs: 

Auditory cue set: 


Necessary depending on the particular application. Method for 
interpolation must be evaluated. 

The signals to be input to the display should be designed to be easily 
localizable: fast attack transient envelope, broad band spectrum. 


Reverberation 

synthesis: 

Coordination with 
visual modality: 


Necessary for externalizing sound sources; could potentially mitigate 
front-back errors. 

Requires further investigation- possible exaggeration of auditory space 
in relation to visual space may be necessary. 
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Virtual Acoustic Displays: Synthesizing Non-Spatial Auditory Cues 


WMe^patial cues are clearly a critical aspect of virtual acoustic displays, very little attention has been 
devoted to what the inputs to spatial sound systems must be. Non-speech sounds can provide a nch 
display medium if they are carefully desiQned with human perceptual abilities in mind. Just as a movie 
with sound is much more compelling and informationally-rich than a silent film, so will a VE be enhanced 
by an appropriate "sound track" to the task at hand. 

One can conceive of the audible world as a collection of acoustic objects .. In addition to spatial location, 
acoustic features such as temporal onsets and offsets, timbre, pitch, intensity, and rhythm, can specify the 
identities of acoustic objects and convey meaning about discrete events (auditory icons representing 
object collisions or feedback for dataglove gesture-recognition), ongoing actions (data sonification/ 
representation of airflow in a virtual wind tunnel), and their relationships to one another. The ideal 
synthesis device would be able to (1) flexibly generate the entire continuum of acoustic features for 
multiple, simultaneous sources, and (2) continuously modulate acoustic parameters associated with these 
sounds in real-time in response to ongoing events. 

State of Knowledge /Technology . 

Some basic principles for design and synthesis of non-spatial acoustic features can be gleaned from the 
fields of music, psychoacoustics, and higher-level cognitive studies of perceptual organization. Recently, 
a few studies have addressed methods for analytically modeling environmental sounds such as propellor 
cavitation, breaking or bouncing objects, and walking sounds, and structurally-based models of source 
characteristics like radiation patterns. 

Current devices available for generating nonspeech sounds are based almost exclusively on MIDI ^ 
(Musical Instrument Digital Interface) technology and tend to fall into two general categories; samplers , 
which digitally store sounds for later real-time playback, and "synthesizers", which rely on analog or digital 
sound generation techniques originally developed for imitating musical instruments. With samplers, many 
different sounds can be reproduced (nearly) exactly, but substantial effort and storage media are required 
for accurately pre-recording sounds and there is usually limited real-time control of acoustic parameters. 
Synthesizers are more flexible in the type of real-time control available but less general in terms of the 
variety of sound qualities that can be generated. Potential disadvantages of both are that they are not 
specifically designed for information display and require specialized knowledge of musical/production 
techniques. A few systems using off-the-shelf devices have been integrated with spatial sound for VEs 
and some designers are developing systems for data visualization or "sonification". 


NdgcIs 

Sound-generation technologies specifically aimed at information display, including generalizable, 
analytical and structural models of source characteristics, need to be developed. More critical need is tor 
research into lower-level sensory and higher-level cognitive factors that allow humans to organize the 
intermixed streams of sound that make up the acoustic world into individual, comprehensible objects. For 
example, there is little or no research available on how humans identify, segregate, and localize more 
than two simultaneous sources. Also, seemingly independent acoustic features/parameters like pitch and 
intensity can interact in unexpected ways so that what was intended to be two sound sources can appear 
to be a single source. Understanding of such effects is critical for designing and using synthesis devices. 


VE System Requirements 
Number of sources 


7 +/- 2 simultaneous sources-short-term memory probably limits the 
number of sources that can be processed at once by human users. 


Multiple-source 

perception 

Flexible synthesis 
techniques: 


largely unknown; perceptual and cognitive studies needed to provide 
design guidelines. 

not yet available; initially "hybrid" sampler/synthesizer systems 
designed for information display can be built; generalizable, analytical 
models of source characteristics need to be developed. 


Real-time parameter 
modulation: 


> 31.25 Kbs; MIDI baud rate inadequate for multiple-source control; 
required speed and number of parameters depends on perceptual 
research outcome. 
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Virtual Acoustic Displays: Issues for Hardware/Software Development 


ISSUG , 

The primary issues for improved 3-D auditory displays lies in digital signal processing (DSP) hardware 
and software, and headphone and head-tracking interfaces. Issues related to the head-related transfer 
function (HRTF), the key data component to the simulation, are particularly relevant. 

Status 

Two primary hardware devices have been developed through Ames to date. The first is the Crystal River 
Engineering Convolvotron family of devices, which are used primarily for situational awareness and 
psychophysical research. These are characterized by (1) utilization of a host computer platform, (2) 
capability of spatializing multiple inputs to anywhere in 3D space, albeit with concurrent degradation of 
HRTF with increasing number of sources; (3) capacity for interpolation; (4) Polhemus head tracker 
interface; (5) analog input and output. The second device, an Ames prototype, is designed for radio 
communications; it uses no head tracking interface or host computer, and is not capable of more than 4 
(selectable) fixed spatial positions. 

Needs 

Since the DSP capability of any device is finite, research into how to reduce the amount of data used to 
represent HRTFs used is very important for (1) designing less expensive virtual acoustic displays, and (2) 
allowing for allocation of DSP resources to issues such as simulation of reverberant environments (cntical 
for source extemalization and overall veridicality). For example, reverberation simulation currently 
requires large scale, expensive hardware, and is designed to approximate physical parameters. 

However, research into identifying the perceptually salient features of reverberation will most likely allow 
relaxation of the of physical modeling parameters, thereby freeing DSP resources for other chores. 
Increasing DSP resources also allows for the possibility of including "custom tailoring" routines into virtual 
acoustic displays, such that users could either 1) alter a display to their preferences for optimal 
localization, or 2) allow measurement and storage of individual HRTFs. The latter will be facilitated by 
redesigning systems around floating point signal processing chips, rather than the fixed point chips 
currently in use. 

The nature of communication and simulation systems will become increasingly digital in all aspects, 
including fiber optic transmission, all-digital mixing and processing, and digital interconnectivity-- analog 
interfaces will exist only at the terminus point of the system. Hence, it is essential that existing devices be 
modified in the future to accommodate the range of digital interfaces currently in use (e.g., AES-EBU). 
This also will require enhancement of devices to take full advantage of the > 90 dB signal/noise ratio 
available on 16-bit digital audio systems. Interface connectivity will also require redesign as tracker 
technology, which is currently in its infancy, develops and offers new methods of transmission. The 
trackers themselves suffer from magnetic field distortion, cumbersomeness, and speed; more often than 
not, the weak link of a virtual acoustic display is the interface between tracker technology and sound 
spatiaiization software. Hence, virtual acoustic displays shall require modification as better trackers are 
developed. For communication system displays, size and portability are important considerations for 
applications such as manned space missions; research into miniaturization and reducing power supply 
demands are essential. Size is also a consideration in applications such as test director consoles. 

Finally, research into stereo headsets acceptable to communication personnel may be necessary 


VE System Requirements 
DSP platform: 

miniaturization 
system interface: 


Should allow (1) data reduced HRTFs; (2) reverberation simulation; (3) 
HRTF measurement capability; (4) real-time floating-point DSP. 

Size and power requirements should be minimized. 

Non-cumbersome head trackers; allowance for digital I/O; stereo 
headsets usable by communication personnel; means for customizing 
HRTFs. 
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Host Computers 


Cifrrent VE host computers range from relatively inexpensive personal computers, with add-on graphics 
and video boards, to expensive, high performance graphics workstations. Most VEs are currently built 
with polygons, so transformation of polygons is the dominant processing capability. The use of texture 
mapsadds visual detail at less computational cost than reliance upon polygons alone, so systern support 
for texture maps is increasing. A high-end system can produce 7000 small, anti-aliased, textured 
Dolyqons per graphics pipeline at 30 Hz. This system can provide 380 128 x 128 simultaneous textures, 
with a maximum texture size of 1024 by 1024 texture elements. Renderers are designed to fill 
workstation screens with images of uniform resolution. Pipelined graphics systems with their inherent 
latency (e.g. 50 milliseconds on one high-end system) dominate the scene, but efforts to reduce this 
latency are making some progress in graphics hardware architecture research labs. Parallelism in most 
VE host computers is limited to, at most, a few parallel processors, but new highly parallel graphics 
architectures are being developed. Massively parallel processors (MPPs), currently the do ^ ir ]°t 
supercomputers (sustained processing rates of a trillion floating point operations per se c»nd are experted 
by the year 2000), are being applied a few very high-end (and very costly) VE systems. .^'^Shmuch 
is being written about fully integrating video and audio into information systems of all kinds, including VEs, 
his effort is far from reaching its goals. Further, image processing systems and graphics systems, whtch 
are slowly coming together in medical VE systems, still remain far apart in design, emphasis, and 
interactivity. 

R6S63rcli Needs 

Human performance research is needed to characterize, to the greatest extent possible, the parallelism 
inherent in user-environment-system interactions, for all target applications, in order to map the prob e 
to the MPP-based VEs of the near future. 

Human performance research is needed to determine the texture map sizes necessary to support user 
visual/informational requirements for applications that will benefit from VE. For example, virtual planeta^ 
exploration requires texture maps that are much larger than those provided by current systems. £ ss 
of the use of texture maps to support planetary exploration systems and users, and similar analyses in 
other applications, will provide essential guidance to VE system architects. 

Human performance research is needed to determine the types of data required for human visualization 
of complex systems and phenomena, and the relationships among those data types, in each application 
domain of interest, as guidance for the design of multimedia VE architectures. 


Systems must provide a capability to texture map planet sized objects at centimeter resolution, if not by 
brute force of massive memory, then by means which give the user the appearance of continuous 
mapping in real-time. 

In order to effectively utilize the hundreds and even thousands of parallel processors of MPPs for VE, 
toolkits based on human performance research must be developed that can extract the parallelism 

inherent in VE applications. 

Renderers must be specifically optimized for use with head mounted displays for VE systems. For 
example rendering based on human vision would put more detail in the center of the image and less 
toward the periphery. When there is no eye-tracking, a "pseudo-fovea" of plus or minus 15 degrees 
containing high detail, with a fall off of detail toward the periphery, would HMD 
use. This is so because humans typically make eye movements of plus 0 ^Thus 

straiaht ahead and will typically turn their heads as necessary in order to look beyond that. Thus, 
rendering to support HMDs is very different than that to support desktop workstations, and must be 

optimized accordingly. 

Lvle^ B (1993) Media spaces and Broadband ISDN. Communications of the ACM, 36(1), 46-47. 
Silicon Graphics, Inc. (SGI). (1992a). Reality Engine: Virtual reality. Data sheet. 

Silicon Graphics, Inc. (SGI). (1992b). Reality Engine: Visual simulation. Datasheet. 

Silicon Graphics, Inc. (SGI). (1992c). Performer manual. 
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Networks 


State of Knowledge/Technology 

Local area networks (LANs) and wide area networks (WANs) are becoming increasing central to 
computing and communications, and there is much research being done to increase network capability 
and accessibility. Distributed and shared VEs, being developed in research labs, will benefit from high 
speed networks. The Broadband Integrated Service Digital Network (B-ISDN), an emerging 
telecommunications standard, will provide user-to-user bandwidths of 10's of millions of bytes per second, 
with total bandwidths of perhaps hundreds of billions of bytes per second. Access to Internet, the world s 
largest computer network, is rapidly increasing. While shared quasi-VE's via Internet do not provide a 
sense of presence, the information sharing capability provided by Internet is already providing forums for 
discussions and announcements regarding VE ("virtual communities in cyberspace"), and collections of 
free VE models and toolkits. Despite these pioneering efforts, use of networks for collaborative shanng in 
support of VE is largely underdeveloped. 

While raw speed and interconnectivity of networks are increasing, to the benefit of VE research and 
development, some difficult organizational and social issues are similarly on the rise. The openness of 
Internet raises competitiveness issues that cause concern to some. For example, some institutions which 
are unaccustomed to, or uncomfortable with, the implications of widespread network communications are 
actively discouraging information sharing on Internet. As access to network resources is increasing, 
many fundamental social issues are also being raised (e.g. copyrights, rights to information access, 
protections of privacy). For example, research into practical matters of dealing with the information glut, 
such as through the use of information filters, is based in part on modeling or profiling the interests and 
access patterns of users. As distributed VEs enable complex user behaviors or highly targeted data 
searches, access to user interaction profiles will have vast implications for marketing, privacy, and 
security. 


Research Needs 

Human performance research is needed to determine the network performance requirements, and the 
implications of communication latencies, for a wide variety of VE capabilities and applications, including 
both distributed VEs and network-supported VEs. 


The lessons learned from research on SimNet (the DOD's distributed battle simulation system) and rts 
successors, as well as from the current research in networking heterogeneous military simulators, need to 
be made readily available to the VE community. This will greatly benefit the utilization of available and 
future network capability for application to general purpose VEs, and foster commercial development. 


VE System Requirements B 

Network-based VE systems require network bandwidths sufficient for establishing a sense of presence 
created from all essential forms of user domain data, including digital models, static and dynamic imagery, 
voice, etc., for a wide variety of applications. 

VE resources such as models, worlds, toolkits, need to be distributed via large area networks. 


Policies of participating institutions regarding access to and use of networks for VE need to be brought 
into alignment with technological capabilities. 

Social issues of networks and VEs must be widely discussed and debated, consensus must be reached, 
system-safeguards must be implemented, and regulations must be legislated. 
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Models, Imagery, and Other Data 

Virtual°enviro^ built of digital models of things and phenomena, along with means to 

represent them to human perception. A room, for example, can be represented as of 

numerically defined geometric shapes. The wind over a wing can be indicated as stnng-like objects 
consisting of short line segments whose endpoints have numerical coordinates^ A photo can be texture- 
mapped onto a shape, or a sound associated with an object behavior. Users themselves cou d be 
extensively modeled within a VE, although only very limited user modeling is currently done. Use of data 
from various disciplines, such as planetary exploration, computational chemistry anatomy, and f uid 
dynamics provides some very realistic models, though these are not always well suited to VEs. Models 
created explicitly for VEs. however, are typically primitive, "hand crafted", and far from i compre ihens. ve in 
their representation of the thing or phenomenon modeled. One reason for this is that the limits of 
available computing power and the need for an acceptable update rates combine to limit the model 
complexity that can be presented. Another reason for the current limitations of VE models ts_ that 
object/environment scanning and computer aided design tools for VE are underdeveloped. The 
integration of the many kinds and formats of models, imagery, and data is hampered by the large number 
of incompatible formats. Although some small companies have begun to provide 3D dip objects 
analogous to 2D clip art, there is no significant marketplace of virtual objects. Some objects and data 
appropriate for VEs are available for free on Internet. Representation and processing of the behaviors 
and other non-geometric characteristics of digital entities in VEs are currently underdeveloped. 

Research Needs ... , 

Human performance research is needed to understand user domain requirements for utilizing 

models/data. This would provide, for example, an understanding of how large terrain ^ 

subdivided. It would also guide the development of methods for real-time traversal of virtual ferrain that 
involves multiple data files. This research would also provide an understanding of how disparate forms of 
data are related to each other by the user in each of the studied user domains. 

Artificial intelligence research is needed to enable automated scanning of physical environments, 
automated differentiation of individual objects, intelligent attribution of object characteristics and 
behaviors, and automatic archiving. 

Computer graphics research is needed that will enable virtual objects to, show the effects sol -use 'Overtime, 
such as wear, and modifications comparable to those available with real world objects. Research is also 
needed to enable these and all other VE model characteristics to be modifiable in real-time. 


Toofs V and?echniqu^ that can rapidly digitize environments and intelligently differentiate 

individual objects. 


Readily accessible collections of hierarchical geometnc and behavioral models are 'needed. ™®se 
collections should be accessible via network. It should be possible to add new models easily and to 
search the archive rapidly for models needed by specific applications. Methods and systems are needed 
to enable archival objects to be modified, customized, and personalized. 


Standards are needed among model formats to enable sharing and to eliminate expensive, ad hoc model 
development. 


Models must have capacity for individualization, and the unique marks of use overtime. 

ft should be possible to integrate all forms of data and models, for example, to map live or recorded video 
with sound onto polygons within a VE. 


Grossman, M. (1992). Modeling reality. Spectrum, 29 (9), 56-60. 
van Dam, Andries. (1992). 1991 Steven A. Coons Award Lecture. 


Computer Graphics, 26 (3), 205-208. 
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Software Tools 


State of Knowledge/Technology i£W _ . .. . .. . . 

Several toolkits are available for the creation and management of VEs. High-end tools such as sei s 
Performer help with real-time simulation by providing a library for optimizing rendering functions, efficient 
graphics state control, and other functions. Performer also provides a visual simulation application 
development environment which provides multichannel and multipipeline capability; parallel simulation, 
intersection, cull, and draw processes; hierarchical scene graph construction and real-time editing; system 
stress and load management; level of detail model switching; fixed frame rate capability; and other 
features. 


University and commercial toolkits are becoming available to support "virtual world creation", including 
Minimal Reality (MR) Toolkit from University of Alberta, WorldToolKit from Sense8, Cyberspace 
Development Kit from AutoDesk, and the IBM Virtual Reality Toolkit. These help to manage the various 
input/output devices, graphical objects, object interactions, and views. Some, like the IBM system, 
support distributed VEs. 

3D “widget” toolkits are being developed in a few university research labs so as to go beyond 2D buttons, 
sliders, and other window-oriented control devices. The 3D widgets are intended to better support the 
virtual reality paradigm. 

Computer aided software engineering (CASE) tools are commercially available to automate some of the 
manual activities of software engineering, to support adopted methodologies, and to improve quahty and 
reliability CASE tools are not yet widely used, however, and there are none specifically for VE. CASE 
tools require significant training, and there is no hard data indicating that they improve productivity. Since 
software development is so costly and complex, however, tools that improve the development process 
have significant appeal, and are undergoing intense development. 


Human performance research is needed that will serve to guide the development of user domain-based 
toolkits. Such research would include analysis of the commonalities among related user applications. In 
addition, joint human performance and computer science research will be needed to integrate graphical 
toolkits and user domain toolkits. 

Human performance research is needed to advance content analysis of domain documentation, including 
voice transcripts, post-mission debriefings, formal reports, photographs, maps, and all other forms of 
domain data. This will provide guidance for the design of computer aided, object-onented analysis toolkits 
to support the creation of application-oriented VEs. 


VE System Requirements . 

Using available toolkits, and network accessible geometric, behavioral, and user domain models, a 
knowledgeable individual should be able to create a useful prototype of an application-based VE in one 
week A team of graphics and user domain analysts should be able to refine the VE for initial use by 
domain users within a month. It should be possible to complete a final production system within 60 days. 
Further, it should be possible to make the necessary modifications to meet changing user requirements 
with a minimum of effort. Ultimately, users should be provided with toolkits that will enable them to 
develop and enhance their own specialized VEs. 
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Domain Analysis and Design 

Us^ do^r^n ais^sfs^id* 1 use °do main based design are critically important for the development of useful 
VEs but are currently underdeveloped. Instead, the current emphasis in most VE work is on graphical 
techniques and hardware integration. There is little if any literature on the spatial, environmental, and 
informational interaction requirements of the many user domains for which VEs are claimed to be needed. 
Further, the role of presence and virtual presence in these user domains is largely unexplored. While 
extensive literature exists regarding remote operations of robots, only recently have the elements of 
operator sensory immersion, presence, and virtual presence been explicitly addressed. Object-oriented 
analysis (OOA) and design (OOD) methods, which are still under development, are being widely 
discussed and refined in the computer science literature. Some VE projects are beginning to apply and 
extend these methods by incorporating field observation techniques derived from ethnographic field 
methods The application of object-oriented techniques promises to provide analyses that can easily be 
incorporated into VE design, and designs which are easily understood, extended, and reused. Traditional 
"task analysis" is related to this activity by the common interest in humans as a component in human- 
machine systems". The emphasis of task analysis, however, is very different from that of the object- 
oriented approach in that task analysis is procedure-oriented, concentrating on actions. By concentrating 
on objects the object-oriented approach seeks to find the relatively stable structure on top of which any 
procedures might be layered. This lends stability and reusability to both analyses and designs that are 
unavailable to a procedurally oriented approach. 


Research Needs 

There is a need for human performance research to analyze and model the informational and sensory 
components of the environments of domain experts (which are strictly relevant to their tasks) and the 
user-environment interactions, for applications that are to benefit from VE systems. These domain 
analyses must provide results in a form that can directly apply to VE system designs. 

There is a need for human performance research to compare and contrast candidate VE user domains so 
as to extract the commonalities that might be profitably addressed by general purpose, commercial VE 
systems and toolkits. These cross-domain analyses must provide results in a form that can directly apply 

to VE system designs. 

The mapping of real-world tasks to VE tasks introduces many degrees of freedom not available to the 
user in the non-virtua! task environment. Accordingly, user domain analysis must not be limited to the 
real-world task as currently performed. Instead, human performance research is needed that provides a 
fundamental understanding of the user-environment relationships beyond the constraints of the current 
techniques, and these must be compared and contrasted to non-VE user-environment relationships, in 
order to demonstrate conclusively whether virtualization can benefit the user. 

There is a need for human performance research to analyze and model the role of presence in proximal 
operations and the utility of a sense of telepresence in remote operations. User-environment 
relationships associated with presence and virtual presence in specific user domains must be analyzed, 
modeled, and documented in a form that contributes to VE analysis, design, and implementation. 

The results of user domain analyses must be structured so that they directly apply to VE 
VE analyses and designs for VE systems must be reusable for a variety of applications related to the 

target domain. 
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Spatial trackers provide the VE system host computer and rendering software with the six degree of 
freedom position and orientation of various parts of the human operator’s body with respect to "rear 
space. Trackers attached to the head permit the correct orientation and positioning of visual or audio 
scenes within the display hardware. Trackers on the hand allow gestural and pointing modes for manual 
interaction with the VE. 


State of Technology 

A number of spatial sensor technologies are currently available in commercial quantities at reasonable 
prices. The first type of sensors are “non-contacting” devices based on measurement of either 
electromagnetic, ultrasound, or optical emissions, where the only encumbrance to the user is the 
electrical cable to transfer power signal from the component worn by the human user. The second type of 
sensors are "contact" devices that couple the body segment of interest to the a fixed point by a 
mechanical linkage that is instrumented by position transducers. 

Some of the drawbacks associated with the various technologies include: 

Cumbersome weight and inertia for some larger portable non-contact sensor components and all 
mechanical linkage devices 

Limited range of travel due to transmitter strength for non-contact technologies and linkage length for 
contact devices. 

Line-of-sight and other obstructions near or between sensor and transmitter pairs— by any object 
optical and ultrasound devices and metallic objects for electromagnetic ones — which can spatially 
distort or even completely eliminate measurements. 

Unexplained systematic distortions in reported spatial measurements that require elaborate and time 
consuming calibration and correction procedures. [1] 

Excessive time delays from on-board processing and data transfer to host computer [2. 3]. 

Noise (i.e., spurious time varying changes) in measurements returned by the sensors, due to 
presence of obstructions or other active energy sources in the “real” environment. 

Filtering, a common approach to noise reduction, however increases time delays [3]. 

VE System Requirements 

If the current dominant sensor technologies are to used effectively and reliably for research and training 
purposes, the shortcomings listed above need to be addressed. At minimum detailed and accurate 
spatial and temporal calibrations need to be available so that tracker users can implement appropriate 
corrective measures, or at least be aware of limitations in data obtained with these devices. 

Other sensor technologies (e.g., inertial and acceleration based schemes) that may not be as susceptible 
to some of the above problems need to be developed. 
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Chapter 3: Brief Summaries of Virtual Environment 
Research and Applications Development at NASA 

This chapter contains brief summaries of current research and development in Virtual Environment (VE) 
technology at NASA. The section is organized by center, and within a center by the name of the principal 
investigator. Descriptions include research objectives, current research status, future research plans and 
system configurations. 
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Dynamic R©spons© of Virtual Environmont Spatial S©nsors 

Principal Investigator: Bernard D. Adelstein 

Co-investigator: Stephen R. Ellis 

NASA Ames Research Center 


Research Objective . 

To characterize the dynamic response of displacement transducers commonly used in VE 

applications. 


A testbed and method for measuring the dynamic response characteristics of position and orientation 
sensors have been developed. The testbed consists of a motorized swing arm that imparts known 
displacement inputs to the VE sensor. The experimental method involves a series of tests in which 
the sensor is displaced back and forth at a number of controlled frequencies that span the bandwidth 
of volitional human movement. During the tests, actual swing arm angle, as determined by an optical 
encoder at the motor shaft, and reported VE sensor displacements are collected and time stamped. 
Because of the time stamping technique, the response time of the sensor can be measured directly, 
independent of latencies in data transmission from the sensor unit and any processing by the 
interface application running on the host computer. Analysis of these experimental results allows 
sensor time delay and gain characteristics to be determined as a function of input frequency. Results 
from tests on several different commercially available VE spatial sensors have been obtained and 
documented. 


Future Plans 

Additional sensors will be tested, 
dynamic response of complete — i.e., 


The test method and hardware will be extended to measure 
end-to-end (sensor-to-display)— VE systems. 


System 

Displays: 

Input: 


Architecture 

Polhemus Isotrak, Polhemus Tracker, Polhemus Fastrak, Ascension Bird, 
Ascension Space Navigator (Big Bird Prototype), Ascension Flock of Birds, 
Logitech Red Baron. Custom built servo-motor swing-arm. 


Rendering: na 

Computation: IBM PC-AT 

Software: In-house 
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Head-Slaved Roll Compensation in Virtual/Remote Target 

Acquisition Tasks 

Principal Investigators: Bernard D. Adelstein and Stephen R. Ellis 
NASA Ames Research Center 


Research Objective „ . . ,. . . 

To examine whether the addition of a head-slaved roll degree of freedom in a computer synthesized 
scene or in a camera platform assists human subjects In judgement of the planar position and planar 
orientation of objects in virtual or remote environments 


It has been suggested that inclusion of the roll degree of freedom (dof) in either computer 
synthesized or remotely viewed scenes may improve situation awareness in tasks such as 
teleoperation. In this work, we examined whether the addition of a head-slaved roll dof in a camera 
platform assists human subjects in judgement of the planar position and planar orientation of objects 
in remote environments 

Six subjects were required to match the position and orientation of a series of stationary target markers 
on a remote taskboard by manually placing similar response markers on an identical local taskboard. 
The subjects could only view the remote taskboard through a head mounted display (HMD) driven by 
the cameras on the platform. They could not see either the local board or their own limbs. The target 
locations spanned the full range of head azimuth for each subject and necessitated near maximal 
head elevation at maximum azimuth magnitudes. 

The data show that the addition of head-slaved roll compensation to the platform had no statistically 
discernible effect on the ability of the subjects to match the position ( le ., azimuth and elevation) of the 
remote targets. Nonetheless, systematic position errors were noted regardless of the roll condition. 
Absence of the roll dof, however, did affect the subjects' judgment of target orientation when their 
heads were at the peak attainable elevations for full magnitude azimuth rotations. 


Further target acquisition experiments requiring head azimuth-elevation-roll will be conducted in a VE. 
The objective of the work will be the development of minimum kinematic degrees of freedom models 
of head motion. 


System Architecture 

Displays: VPL EyePhone HMD. 

Input: Acension Space Navigator (Big Bird prototype). 

degree of freedom stereo camera platform. 


Rendering: na 

Computation: IBM PC-AT 

Software: In-house 


Fake Space Molly three 
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1. Adelstein, B.D., and Ellis, S.R. (1993, October). Effect of Head-Slaved Roll Compensation on 
Remote Situation Awareness. To be presented at 37th Annual Meeting of the Human Factors and 
Ergonomics Society, Seattle WA. 
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Interface Control Parameters for Effective Haptic Display 

Principal Investigators: Bernard D. Adelstein 
Co-investigators: Louis Rosenberg (Stanford University) 

NASA Ames Research Center 

Research Objective 

To understand the mechanical impedance and digital (i.e., discrete time) control requirements for 
effective presentation of haptic information through force reflecting interfaces. 


A haptic or kinesthetic virtual environement VE incoiporates two principal components— the 
computer model (i.e., algorithms or equations) describing the physical dynamics of the objects in the 
VE and the interface hardware that allows the human to interact with the VE. Because a haptic 
interface uses the same piece of hardware serves as both control output and sensory input for the 
human operator, information transfer entails significant bidirectional mechanical power flow. "Huis to 
optimize information transfer to the human, physical power flow must be modulated by selecting the 
appropriate impedance (i.e., mechanical characteristics) for the interface. 

In this work we are conducting psychophysical experiments to see how interface computer control 
parameters affect haptic perception. Parameters of interest are controller sample-and-hold update 
rates and phase lag — features akin to screen refresh rates and persistence in CRT video displays. 


Future Plans . .. 

Examine how intrinsic mechanical properties of the interface such as friction, inertia, and compliance 

affect haptic perception of virtual objects. 

System Architecture 

Displays: Custom built two degree-of-freedom force reflecting joystick 

Input: Custom built two degree-of-freedom force reflecting joystick 

Rendering: na 

Computation: i486DX-50 

Software: In-house 
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3D Auditory Displays in Aeronautical Applications 


Principal Investigator: Durand Begault 

Co-investigator: Elizabeth Wenzel 

NASA Ames Research Center 


Research Objective , .. . ... „ ... 

The purpose of the research is to implement and test auditory display concepts which will allow a pilot, 
crew member, or ATC controller to immediately, accurately, and inexpensively monitor three-dimensional 
information, such as traffic location, through the use of a virtual acoustic display. 


StBtUS 

Hardware has been designed and implemented for experiments in the following areas: spoken audio 
waminq system signals, TCAS (Traffic Collision Avoidance System) advisories, and cockpit radio 
communications. The TCAS traffic advisory consists of the spoken word Traffic". The actual position of 
the traffic is usually obtained visually, through instrument monitoring and/or out the window acquisition, in 
the ACFS flight simulator, the out-the-window position of the traffic is linked to the virtual auditory position 
of the word Traffic” heard through headphones. A pilot experiment was recently completed to determine 
if the time interval for traffic acquisition is reduced when binaural sound delivery is used to suggest the 
direction for head-up visual search of the target, compared to monotic (single ear piece), normal practice 
conditions. Results showed that there was a 2.2 second improvement in acquisition time. An experiment 
currently under development pits a normal TCAS visual display against a 3-D auditory and visual head-up 
display For communications, a 3-D sound hardware system can place various radio communication 
streams (e.q. ATIS, ATC.VOR) in separate virtual auditory positions around the pilot. The purpose of the 
system is to allow greater intelligibility against noise, or in situations where more than one frequency must 
be monitored, such as in the vicinity of an airport. Preliminary investigations have shown a 6-7 dB 
intelligibility improvement over monaural systems. 

Basic research in the area of human headphone localization is also underway that supports these applied 
research efforts. A study was completed that compared inexperienced listeners' headphone localization 
of speech to previous studies using noise. Another study was completed that used artificial spatial 
reverberation to increase the veridicality of the 3-D sound display. Another study determined that there 
was no perceptual degradation when using a 20:1 data reduction of the filter parameters used in the 3-D 
sound display hardware. 


Future Plans 

Additional experiments involving target acquisition and speech intelligibility are planned. 


System Architecture 

Displays: Stereo headphones. 

Input: Polhemus, 4 analog sound sources 

Rendering: Convolvotron, Yamaha TX-16W 

Computation: 386 PC, Mac II cx (non-real time) 

Software: Custom. 
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The Virtual Windtunnel 


Principal Investigator: Steve Bryson 

Co-investigators: Creon Levit, Michael Yamasaki 

NASA Ames Research Center 


Research Objective 

To make possible the visualization and exploration of three-dimensional CFD solution datasets in a natural 
and efficient manner. To explore the applicability of VE technology to numerical flow visualization. 


A VE for the exploration of three-dimensional numerically generated flow fields has been implemented.^ 
The analogy is to a wind tunnel, with the user able to move freely about the flow, injecting "virtual smoke 
into the flow to make it visible, and yet not disturbing the flow in any way. The current environment allows 
the exploration of large single-grid three-dimensional steady flow fields or small single- grid three- 
dimensional unsteady flow fields. The environment supports several numerical flow visualization 
techniques including streamlines, streaklines, particle-paths, and tufts. Collections of tools that generate 
these flow visualizations may be repositioned or reoriented in the flow in real-time. Time can be frozen for 
detailed exploration of complex spatial structures. A new version of the software has been written that 
uses a convex minisupercomputer for memory-intensive and compute- intensive tasks, but still uses the 
IRIS workstation for rendering and display. 


The system is being extended to handle a larger class of engineering flows. This involves support for mul- 
tiple zone grids and large disk-resident unsteady flows on the convex. The software is also being 
enhanced to support multiple users exploring the same flow together in real time. A new generation of 
hardware, namely the Fake Space Labs BOOM 1 1C high-resolution two-color display and Silicon Graphics 
dual-pipe Skywriter are being integrated into the virtual windtunnel. 

System Architecture 

Displays: Fake Space Labs BOOM II and BOOM lie 

Input: VPL dataglove model 2, Polhemous, keyboard, mouse 

Rendering: Silicon Graphics 380 VGX 

Computation: Silicon Graphics 380 VGX, Convex C-240 

Software: In-house 
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Measurement and Calibration of Static Distortion of Position 

Data from 3D Trackers 

Principal Investigator: Steve Bryson (Computer Science Corporation) 

NASA Ames Research Center 


Research Objective , ._ 

To characterize the accuracy of the position sensitivity of trackers commonly used in VE systems. To 
study various methods for correcting the deficiencies in their accuracy. 

Accomplishment Description .. 

Three-dimensional trackers are becoming increasingly important as user inputs in interactive computer 
systems These trackers give the position of a sensor in three-dimensions. If the tracker were perfect, the 
position returned by the tracker would exactly correspond to the position of the sensor In appropriate 
coordinates. In reality, trackers fail to be perfect. Distortions are introduced into the tracking data so that 
the position returned by the tracker only loosely corresponds to the actual position of the sensor, and 
then only with a limited volume of space. This distortion is typically a function of the sensors distance from 
some source, and is dependent on the ambient environment. If this distortion is constant in time, it can be 
measured and the actual position of the sensor can be inferred from the distorted data. This is called 
calibrating the tracker. 

Detailed measurements have been madeof the tracker output for a known set of tracker positions. These 
known tracker positions fill a volume of space with a resolution of 12 inches. Using these measurements, 
the distortion of the tracker data has been determined. Calibration methods that partiaslly correct for the 
distortion have been implemented. These measurements have been performed on a Polhemus Isotrack 
tracker, an electromagnetically based tracking system that provides three-dimensional position and 
orientation of a sensor. The measurements have been taken twice within the same location and at 
different locations to measurement the dependance of the distortion on the ambient electromagnetic 

environment. 

Study of the noise and repeatability imply limits on calibration success. When tracking greater than 50 
inches from the source, the tracking signal is so noisy that no useful calibration can be expected. Inside 
th?s distance, repeatability implies a limit of about an inch in calibration accuracy. Both 4th order polynomial 
calibration and bump lookup calibration perform close to these limits. 


Further study in the calibration question can proceed in two obvious directions. The success of 
polynomial and lookup calibrations suggest that a three-dimensional spline calibration, a combination of 
global polynomials and lookup tables, should work quite well. Lookup calibration requires more 5 
the weighting and interpolation question. The bump lookup calibration method fails for overly distorted 

data sets and can be refined. 


This research addresses only static position calibration. These trackers also produce orientation data, and 
the study of distortion in the calibration should be performed. 


1. Bryson S. (1992). Measurement and Calibration of Static Distortion of Position Data from 3D Trackers. 
In S. Fisher (Ed.), Stereoscopic Displays and Applications 3. San Jose, CA: SPIE. 
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Virtual Spacetime 

Principal Investigator. Steve Bryson (Computer Science Corporation) 

NASA Ames Research Center 

Research Objective , . ...... —^ 

To make possible the visualization and exploration of the curved spacetime of general relativity. To 

explore the applicability of VE technology to numerical relativity 

Status 

A VE for visualizing the geometry of curved spacetime by the display of interactive geodesics has been 
implemented. This technique displays the paths of particles under the influence of gravity as described 
by the general theory of relativity, and is useful in the investigation of solutions to the field equations of 
that theory. A boom-mounted six degree of freedom head position sensitive stereo CRT system is used 
for display. A hand position sensitive glove controller is used to control the initial positions and directions 
of geodesics in spacetime. A multiprocessor graphics workstation is used for computation and rendering. 
The system has been tested by visualizing the several well-known solutions to Einstein's field equations 
using a variety of techniques. Currently the system has been implemented only for spacetimes whose 
geometry data are available in closed form formulas and for spacetimes whose data are on simple static 
computational grids. While this work is intended for researchers, it is also useful for the teaching of 
general relativity. 

There are interesting spacetimes whose metrics are available as exact formulas. Incorporation into virtual 
spacetime should be straightforward. Examples include the classical Godel solution Bianchi type IX 
cosmological solutions, and a metric describing collapsing dust or photons. A more difficult problem is 
including the results of computational spacetime simulations such as those describing colliding biac 
holes collapsing stars, and gravitational waves. There has been considerable interest expressed by the 
computational spacetime group at the National Center for Supercomputing Applications (NCSA) in 
Urbana, Illinois in using virtual spacetime to view the results of their computations. We are currently 
developing a collaboration for this purpose with Ed Seidel and Larry Smarr. Problems addressed by this 
collaboration include: the meaning of the coordinates used for the numerical simulation and how these 
coordinates should be mapped into the virtual space, developing an interpolation scheme to compute 
geometry data from the computational grid that is fast enough to allow real-time computation of geodesics; 
and managing the very large amounts of data that are the products of these unsteady spacetime 
computations. The data and speed problems are similar to those that arise in the virtual windtunnel 
Distributing the computation to a supercomputer over a high-speed network may be necessary. 

System Architecture 

Displays: Fake Space Labs BOOM II and BOOM lie 

Input: VPL dataglove model 2, Polhemous, keyboard, mouse 

Rendering: Silicon Graphics 380 VGX 

Computation: Silicon Graphics 380 VGX, Convex C-240 

Software: In-house 
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Human Performance in Virtual Environments 

Principal Investigator: Stephen Ellis 

Additional Investigators: M. Tyler, W. Kim, L. Stark (UC Berkeley) 

NASA Ames Research Center 


Research Objective 

Using standardized human manual control tasks, develop and assess techniques to study perceptual 
phenomena that depend upon presentation of a convincing, visually enveloping, spatial illusion 


The manual control task that has been studied in a VE is a version of a three-dimensional tracking task that 
has been used extensively in the study of human performance with panel mounted, three dimensional 
displays. In the current work this task has been used to measure human manual control plasticity as display 
control misalignments were introduced between the head-mounted display and the coordinates of a 
hand-mounted 6-degree of freedom position tracker. The subject's basic task was to move the position 
sensor on his/her hand to cause a virtual 3D cursor, viewed via the stereo, head-mounted displays, to 
track a small target that moved irregularly in three dimensions in front of the subject. In contrast to some 
results in the tracking literature, subjects demonstrated the capacity to learn to perform this task with 
display control misalignment ranging from 0 degrees to 180 degrees. With several hours practice during 
one day, subjects could learn to perform the task with nine different display control misalignments. [1] 

The perceptual phenomenon studied was the influence of visual stimuli on the perception of grav’ity-refer- 
enced eye level. It has been used to measure the fidelity with which a VE system produces a simulation of 
three-dimensional space. Experiments have been performed in a VE and compared with results obtained 
in a corresponding physical environment. A virtual pitched array was produced which was geometrically 
identical to a corresponding physical stimuli. The virtual array had a smaller influence on perceived eye 
level than did the pitched physical array. Measurement of the degree to which the pitched optical array 
influenced the subject's sense of gravitationally referenced eye level may be taken as a measure of the 
completeness of the enveloping spatial illusion. Addition of several grid patterns to the virtual pitched 
array increased the influence of the virtual optic array and indicates the specific type of grid that may be 
optimal to improve the completeness of the enveloping illusion in a VEs. [2] 


Future Plans „ 

Work will be continued utilizing higer frame rates attainable with SGI Skywriter system. Additonal displays 
will be evaluated. 


System Architecture 


Displays: 
Input: 
Rendering: 
Computation: 
Software: 


Custom LCD headmount 
Dataglove, Joystick 

HP 9000 
Custom. 


1. Ellis, S.R., Tyler, M„ Kim, W.S., and Stark, L. (1991). Three dimensional tracking with misalignment 
between display and control axes (SAE Technical paper #91-1390). SAE. 

2 Nemire K. and Ellis, S.R. (1991, November). Influence of pitched visual frame on gravitationally refer- 
ence eye level in a VE. Paper presented at the meeting of the Psychonomic Society. San 
Francisco, CA. 
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Extravehicular Activity Seif-Rescue in Virtual Environments 

Principal Investigator: Stephen Ellis 

Additional Investigators: A. R. Brody (Sterling), R. Jacoby (Sterling) 

NASA Ames Research Center 

Research Objective 

The Extravehicular Activity Self-Rescue in Virtual Environments project provides a current example of a 
practical test of the utility of VE personal simulators. 

Status 

By providing a VE simulation with the mass and moments of inertia of an EVA crew-person, and thruster 
characteristics for a rescue device, researchers are able to examine human performance in a rescue. 
Measurements such as onset time until response, time and fuel necessary to cease rotations, and time 
and fuel required to return to the station may be made for an assortment of failure scenarios. Thrusters 
may be altered in magnitude, capacity, moment arm, and number to examine the effects these parameters 
might have on a self-rescue capability. Different control modes such as pulse, displacement proportional, 
force proportional, and on/off could be compared to determine which works best in terms of fuel, time, 
safety, or any other desired cost function. 

So far, one study was performed in the VE simulator of the Advanced Displays and Spatial Perception 
Laboratory. Simulations were conducted to assess the feasibility, and quantify the fuel and time 
requirements for a stranded crew-person to return himself to a space station after an accidental separation. 
A hand-held thruster, similar to the Hand-Held Maneuvering Unit (HHMU) from the Gemini Program was 
used for propulsion. Virtual environment simulators were determined to be useful for simulating 
accidental separations and provided preliminary evidence that a hand-held thruster is a viable alternative 
for accomplishing a self rescue. Simulation fidelity and validity remain to be established 


Future Plans . , . „ 

Further work will be directed towards measuring the realism of the VE simulation and relating these 
measures towards transfer of training to simulated real environments. 


System Architecture 

Displays: Custom LCD headmount 

Input: Dataglove, Joystick 

Rendering: ISG 

Computation: HP 9000 

Software: Custom. 


1. Brody, A. R., Jacoby, R. and Ellis, S. R. (1991). Simulation of Extra-Vehicular Activity (EVA) Self 
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Telerobotic Planning and Operational interfaces 


Principal Investigators: Stephen Ellis 
NASA Ames Research Center 

Research Objective ... . 

The oaths that robotic mechanisms trace during their missions are subject to numerous quantitative and 
qualitative constraints. While algorithms exist to satisfy these constraints automatically, these techniques 
are often slow, inflexible, idealized, or incomplete. Consequently, there is a need to be able to visualize 
the robot's planned trajectory and to interactively edit it. The editing may be required dunng debugging of 
automatic solutions or used during actual collaborative mission planning when a human user interacts with 
an algorithm. 

St at u s 

Such interaction requires a suitable communication medium. Virtual environments produced through 
head mounted displays coordinated with simulations of robotic manipulators are such a medium [1], When 
enhanced by dynamic or Kinematic simulations in the form of "geometric spreadsheets [2] , such media 
can extend human planning capabilities into new realms. One key benefit is that activities can be planned 
within the simulation and avoid the difficulties presented by long time lags associated with communication 
to distant robotic worksites. They additionally promise to add flexibility and speed to the planning process. 

A VE created through kinematic simulation of a PUMA robot arm has been completed in the Advanced 
Displays Laboratory and connected through an EtherNet to a corresponding PUMA arm in a remote 
laboratory of the Intelligent Mechanism's Group using TO A control [3]. Bidirectional video links have also 
been established. Gesture planning software is currently being developed to plan sequences of robotic 
movements. A initial movement macro definition capability has been developed and demonstrated and 
icons representing potential constraints on motion, such as the forces and torques at the end effector 
have been developed. Alternative menu control techniques are being developed for interacting with the 
macros and the VE itself and have been studied through psychophysical and biomechanical techniques. 

Deve[opment n of VE interfaces for robot manipulators to generally advance the virtual environment 
technology of planning of telerobotics tasks. This will address issues functionally . . 

manipulation and control of virtual objects in a large variety of application areas such as assembly training, 
teleoperation, or laparoscopic surgery. Software will be upgraded to run on the new SGI Skywriter dual- 
pipe graphics system. 

System Architecture 


Displays: 
Input: 
Rendering: 
Computation: 
Software: 


BOOM 1 , PUMA Arm 
Dataglove, Joystick 
icr: 

HP 9000 
Custom. 
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Using Virtual Menus in a Virtual Environment 

Principal Investigators: Richard Jacoby (Sterling Software) 

Stepehn Ellis 

NASA Ames Research Center 

SeveraUspects^VEs^ake menu interactions different from interactions with conventional menus. We 
are developing the features and interaction methodologies of different versions of virtual menus. 

Status 

Teleop the robot teleoperation application, has been the VIEW lab’s primary testbed for different menu 
implementations. This application has been used by the first author hundreds of times and by others 
several dozen times over the last year and a half. Virtual menus have also been part of demonstration 
applications in the VIEW lab for about three years. During that time, well over two hundred people have 
tried these programs and experienced VE. During the development, demonstration and use of t es 
applications, we have had the opportunity to observe the interactions between users and our menus. 

The results of informal observation and discussion have lead us to identify weaknesses in the initial 
version of the menus, and to develop a newer version that addresses those weaknesses. 

Until this fall, the interactions with the VIEW menus have been performed entirely through hand 
gestures and pointing. This has been due, in part, to our hardware configuration: an electronic glove; and 
tracker that the user already wears to interact with the environment in other ways. In addition, we wanted 
to evaluate the premise that gesturing is a natural and efficient method of interacting with l h ® 1 hP 
The VIEW menus are text based. They are very thin rectangular 3-dimensional objects that can be 
displayed at any position and orientation in the environment. Highlighting is accomplished by p»«ng a 
dimly fit rectangle between the menu and the highlighted text. Menu interactions are 1 Jl® 

making a specific hand gesture. Before the menu appears, the user sees a model of a hand and hngers at 
the position and orientation of his real hand and fingers. In the newer version of the menus, a pointer and 
cursor also become visible. 

Although the current menu version is substantially easier to operate than its predecessor, many Improve- 
ments to menu interactions can still be made. . . 

One possible improvement to the current interaction method would be to make the cursor 
“sticky". Another variation on the current paradigm is to not render the graphics mode! of the hand ^nng 
menu interaction. Feedback for pointing would still be provided by the ray and cursor. An advantage may 
be a faster running simulation which would make pointing easier because of reduced transport de ay. 

A different interaction paradigm that has already been partially implemented entails the use of a 
two-button hand-grip that is held in the user's left (ungloved) hand. The buttons are used to 
menus and cycle through menu items. It is hypothesized that this type of interaction may picker t a 
pointing to a menu item. Another advantage of this paradigm is that the user's right (gloved) hand will be 
free for other uses. An experiment is being be designed to test the hypothesis. 

Further investigation is necessary to evaluate how the menu's frame of reference should be 
related to the user’s frame of reference. Work is also needed in the area of gesture re f° 9 n >t'O n - A ^ay 
will be performed to determine how well the computer recognizes gestures and how well it can distinguish 

between the user's gestures. 

System Architecture 

Displays: Custom LCD headmouted display 

Input: Polhemous, Dataglove, hand-held button 

Rendering: ISG 

Computation: HP 9000 

Software: Custom. 

? e jacoby C R. S and Ellis S. (1992). Using Virtual Menus in a Virtual Environment. In S. Fisher (Ed.). 
Stereoscopic Displays and Applications 3. San Jose, CA: SPIE. 
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Presence in Natural Terrain Environments 

Principal Investigator: Michael W. McGreevy 

NASA Ames Research Center 


Planetary geologists assert that human presence is essential to planetary field geology but the nature of that 
presence is not sufficiently characterized to influence exploration systems and mission operations. 


Objective 

The objectives of this activity are to: (1) enhance the effectiveness of planetary surface explorers by extending t 
reach of human presence: (2) understand human exploration behavior, and the nature and benefits of presence 
within natural terrain environments; and (3) broaden the approach to VE research through complementary 
investigations in natural environments. 

Approach 

This field component of the Virtual Planetary Exploration program approaches the question of presence, 
actual and virtual, through f ield studies in which experienced planetary scientists are observed as they 
explore natural environments. Methods of analysis derived from ethnographic and object-oriented 
analyses, which are based on traditional methods of analyzing complex systems, are being developed and 
applied These analyses are being used to model human-terrain interactions characteristic of presence, 
and to guide user-based object-oriented design of virtual planetary exploration systems. A small element 
of this work involves some depth-first analysis of Apollo 17 surface operations^ The Apollo missions offer 
a considerable amount of uniquely valuable information that make it an appropriate area for more 
comprehensive analysis efforts, once the necessary tools are more fully developed. 

Accomplishments . ...... ^ , ... 

Two field studies have been conducted, one to the Amboy lava field in the Mojave Desert, and another to 
the Kaupulehu lava flow of the Hualalai Volcano in Hawaii. The Amboy lava field is a landscape which is 
analogous to terrain on Mars. Two planetary geologists were interviewed and observed there during the 
conduct of typical surface operations. Each subject then wore a head-mounted video camera/display 
system, which replaced his natural vision with video vision, to conduct further surface explorations. The 
study brought to light factors influencing field exploration behavior and continuity relationships among 
them, which have contributed to the elaboration of some initial elements of a theory of presence. It also 
helped to illuminate methodological and theoretical issues of ecological task and site validity. The second 
field study was conducted on the Kaupulehu lava flow of the Hualalai Volcano on the island of Hawaii. Two 
planetary geologists were accompanied on a multi-day geologic field trip that they had arranged for their 
own scientific purposes. The subjects were observed and videotaped during the course of their work, 
and interviewed regarding their activities. Analysis of the Hualalai field activities, related documentation, 
and interviews, using the ethnographic and object-oriented methods that are being developed, is 
contributing to the development of components of a theory of presence by revealing or confirming the 
nature of redundancies in the sense of presence. Further, the analysis is providing feedback useful for 
improving the analysis methods themselves, and is providing detailed domain structure on which to base 
improved designs for user-based, object-oriented virtual planetary exploration systems. 

PlansTnclu'deM) completion of the Hualalai series of field studies: 2) further development of ethnographically 
informed object-oriented analysis capabilities; 3) investigation of related field activities and extraction of 
commonalities; 4) elucidation of exploration behavior and related aspects of presence. 

R&forQncQS 

McGreevy, M.W. and Stoker, C.R. (1990). Telepresence for planetary exploration \n Proceedings of the 
SPIE 1990 Conference on Cooperative Intelligent Robotics in Space, SPIE Vol. 1387, 1 110-1 23. 
McGreevy, M.W. (In press). Presence of field geologists in Mars-like terrain. Presence, Cambridge, MA: 

MIT Press. 
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Disparate Data Integration for Model-Based Teleoperation 

Principal Investigators: Michael W. McGreevy 

Cindy Ferguson (Sterling Software) 

NASA Ames Research Center 


Planetary geologists say that they expect VEs to be especially useful for exploration when digital models 
of terrain elevation and brightness are augmented with spatially correlated data from many sources, 
including: geologic, seismic, biologic, mineralogical, and traversibility maps; diverse static and dynamic 
imagery; time-varying data; volumetric data; archival data; real-time data from on-site exploration systems; 
and ad hoc or preliminary observations of colleagues. 


The objectives of this activity are to: 1 ) enhance the effectiveness of operators and analysts of planetary 
exploration missions by improving the integration of disparate, spatially correlated environmental data; (2) 
develop an object-oriented architecture for integrated exploration of disparate spatial information; and (3) 
support model-based exploration and teleoperation. 


r\ U U I v U v 1 1 m 

The approach taken is to apply and refine object-oriented analysis, design, and implementation 
techniques. The project is intended to translate field, document, and other analyses to user-based, 
object-oriented designs. This involves analyses, coordinated with field studies, that are specifically 
related to the disparate nature of the spatially correlated experience and data. Also involved are analyses 
of the implementation environment in which the domain requirements are to be addressed. Analyses are 
iteratively transitioned into object-oriented designs. Currently, research is addressing a core of 
commonality among several related domains, including field geology at the Hualalai Volcano, Apollo 
surface traversals, and the scientific exploration of Monterey Canyon in Monterey Bay. The intent of this 
core commonality approach is to ensure that designs are robust, generalizable, and reusable. The most 
comprehensive design and implementation will focus on exploration of Monterey Canyon, since there is a 
readily accessible cadre of domain experts already cooperating with this project. 


An iterative, layered, and object-oriented approach has been developed and documented. The 
architectural design has already gone through several iterations which have made fundamental 
improvements without great expense— time, effort, or funds— because the changes occurred well pnor to 
implementation. Initial analyses of several user domains, the implementation environment, and the R&D 
environment are in progress, and are contributing significantly to the approach and the design. A broad 
domain survey, done previously, has provided outlines for the early design effort. Ethnographic and _ 
object-oriented analyses are being developed and integrated. Strategic, narrow, depth-first user domain 
analyses and implementation environment analyses are being done in order to further orient the early 
design effort. Coding has begun on basic software components that are independent of comprehensive 
analyses. A gesture recognition subsystem has been developed in C++. It is being made compatible with 
SGI's high-performance simulation toolkit, Performer. Detailed domain analyses are in progress. 


Plans include: 1) develop a user-based, object-oriented, VE architecture for integrated exploration 
utilizing disparate, spatially correlated data; 1) develop the object-oriented design based on user domain 
analyses; 3) ensure that the design and implementation possess a core capability that supports 
fundamental commonalities of a wide range of related applications; 4) focus a reasonably comprehensive 
implementation on a user domain in which diverse data and users are readily available 


References 

Ferguson, C. and McGreevy, M.W. (1992). NAM AKA: A prototype for disparate data integration for model 
based teleoperation. Unpublished initial design document. , ...... 

McGreevy, M.W. and Ferguson, C. (1993). Presence and virtual presence for planetary exploration. _ 
Program definition , current elements , and plan for desired improvements . Unpublished document. 
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Virtual Planetary Exploration Testbed 

Principal Investigators: Michael W. McGreevy 

Lewis Hitchner (Western Aerospace) 

Co-Investigator: William Briggs (San Jose State University Foundation) 

NASA Ames Research Center 


Problem , . , 

NASA's operational experience in planetary exploration indicates that spatial visualization of human scale 
planetary surfaces is critically important to the success of manned and unmanned missions. Systems 
available for this purpose have very limited capability and users want improvements. 

Objective 

The objectives of this activity are to: (1) contribute to the success of planetary exploration missions by 
improving the operational effectiveness of human and machine explorers; (2) extend the reach of human 
presence and democratize space exploration; and (3) develop concepts and systems for generating and 
testing user-based virtual planetary environments. 

Approach 

Consideration of the needs of the ultimate users is paramount. Fundamental to the VPE approach is the 
use of actual planetary terrain data from NASA exploration missions. Further, use of data from engineering 
tests of future rover imaging systems has been central to development of the VPE Testbed. This 
provides a necessary reality check for concepts and methods. The shear magnitude of the terrain data 
sets is itself a significant factor in the interaction between the user and the VE. So, rather than develop 
VEs based on toy worlds (a common practice), the VPE Testbed activity directly addresses the essential 
conflict between data complexity and user-environment interactivity. 

Accomplishments 

The VPE Testbed can be used to virtually explore digital terrain models (DTMs) of Mars using Viking 
orbiter data, panoramic surface images from Viking orbiter and lander data, and high resolution DTMs from 
sites on Earth. Several techniques have been developed, including a digital version of the Surveyor 
Mosaic Spheres and virtual interactive flight over digital terrain models of variable resolution. Algorithms to 
interactively and non-uniformly vary terrain model complexity, based on user interest, have been 
developed and implemented. Several hand gesture recognition algorithms, and a recognition 
subsystem have been developed. The recognition subsystem, written in C++ on an SGI Indigo, is being 
ported to the SkyWriter and made compatible with SGI's high-performance simulation toolkit. Performer. 

Plans include: 1) enhancement of user interactions with human scale terrain data; 2) further development of 
complexity msnagement slgorithms and software that will provide additional user-based criteria for adaptively 
modifying model complexity to that which is essential to the user and task at any given moment. 

References 

Ferguson, C., McGreevy, M. W., and Hitchner, L. (1993). User Guide to the Virtual Planetary Exploration 
(VPE) Testbed. Unpublished documentation. 

Hitchner, L. (1992). The NASA Ames Virtual Planetary Exploration Testbed. Presented at IEEE Wescon, 

Hitchnef L^nd^/lcGreevy, M. (1993) "Methods for User-based Reduction of Model Complexity for Virtual 
Planetary Exploration.” Proceedings of the IS&T/SPIE Symposium on Electronic Imaging Science and 

McGreevy, M.^W. (1989). Personal Simulators and Planetary Exploration Plenary address presented at 
CHI '89 

McGreevy, M. W. (1991). Virtual Reality and Planetary Exploration. On Line, *3(8), 1,3,8. 

McGreevy, M. W. (1993). Virtual Reality and Planetary Exploration. In A. Wexelblat (Ed.), Virtual Reality. 

Applications and Explorations. Cambridge, M A: Academic Press. 

Musgrave, F. K. (1992). A panoramic virtual screen for ray tracing. In D. Kirk (Ed.), Graphics Gems III. Boston, Mi 

Academic Press. 
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Simulating Complex Virtual Acoustic Environments 

Principal Investigator: Elizabeth Wenzel 

Co-Investigator: Durand Begault 

NASA Ames Research Center 

Research Objective . 

Investigate the underlying perceptual principles of auditory displays and develop an advanced signal 
processor (the "Convolvotron") based on these principles. 

Status 

Spatial auditory displays require the ability to generate localized sound cues in a flexible and dynamic 
manner. Rather than use an array of speakers, the Convolvotron maximizes portability by synthetically 
generating three-dimensional sound cues in realtime for delivery through earphones. Unlike 
conventional stereo, sources are perceived outside the head at discrete distances and directions from the 
listener. This is made possible by numerically modeling the effects of the outer ears (the pinnae) on the 
sounds perceived at various spatial locations. These 'Head-Related Transfer Functions’ (HRTFs) can then 
be applied to arbitrary sounds (voices, for example) in order to cause them to seem spatially located. The 
Convolvotron, a set of two printed circuit (PC) boards hosted by a personal computer, is capable of 
synthesizing up to four simultaneous, localized sources, and implementing smooth motion trajectories in a 
head-stable environment. It is currently being used in a variety of government, academic, and industrial 
laboratories in addition to the Ames Spatial Auditory Displays Lab. 


While the initial implementation simulates only the direct paths of up to four virtual sources to the listener, ft 
possesses a high degree of interactivity. In a simple anechoic space, it is possible to freely manipulate 
source position, listener position, and listener orientation in a dynamic, real time display. It is desirable to 
be able to achieve the same level of interactivity in more complex acoustic environments. 

Psychoacoustical research suggests that synthesis of purely anechoic signals can result in perceptual 
errors, in particular an increase in front-back reversals and a failure of externalization. There is reason to 
believe that such errors can be alleviated by providing more complex cues resulting from reverberant 
environments. 


Of particular interest here is the work on image models for simulating room characteristics with synthetic 
early reflections using a kind of ray-tracing approach. For example, the walls, floor, and ceiling in an 
environment are simulated by using HRTF-based filters to place the "mirror image" of a sound source 
behind each surface to account for the specular reflection of the source signal. The filtering effect of 
surfaces such as wood or drapery can also be modeled with a separate filter whose output is delayed by 
the time required for the sound to propagate from each reflection being represented. 

In future work, we plan to extend this simulation to more realistic models of acoustic environments. Some 
of the issues that need to be addressed are diffuse reflections, scattering reflectors, diffraction and partial 
obscuration by walls or other objects, near-field effects (e.g., head-shadowing), and perceptually-viable 
methods of simplifying the synthesis technique. 


System Architecture 

Displays: Stereo headphones. 

Input: Polhemus, 4 analog sound sources 

Rendering: Convolvotron 

Computation: 386 PC 

Software: Custom. 

References 

Begault, D. R. (1991). Perceptual effects of synthetic reverberation on 3-D audio systems. Journal of the Audic 
Engineering Society, 40, 895-904. . 

Begault, D. R. (1992). Binaural Auralization and Perceptual Veridicality. Audio Engmeenng Society 93ra 
Convention, Preprint No. 3421 (M-3). 

Wenzel, E. M. (1992). Localization in virtual acoustic displays. Presence, 1, 80-107. 
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Hearing Through Someone Else's Ears 

Principal Investigator: Elizabeth Wenzel 

Co-Investigator: Durand Begault 

NASA Ames Research Center 


Research Objective u . 

Synthesis of virtual sources involves the digital filtering of sounds using filters based on acoustic Head- 
Related Transfer Functions (HRTFs) measured in human ear-canals. In practice, measurement of the 
HRTFs of each potential user of may not be feasible. Thus, a critical research question is whether 
inexperienced listeners can obtain useful directional information in a virtual acoustic display without 
requiring that the cues be individualized for each potential user. 

Ststus 

Early experience has suggested that using non-individualized HRTFs is possible, so long as the filters that 
are used were originally measured for someone with accurate localization in both real world and 
synthesized conditions. This study represents a more formal test of this hypothesis. Sixteen blindfolded 
listeners iudqed the apparent spatial location (azimuth [left-right] and elevation [up-down] ) of stationary 
sounds presented either over loudspeakers in the free-field (a non-reverberent environment) or over 
headphones. The headphone sounds were synthesized using HRTFs from a "good localizer” in a 
previous study. 

In aeneral, the data suggest that most listeners can obtain useful directional information froma spatial 
auditory display without requiring the use of individually-tailored HRTFs, particularly for the dimension of 
azimuth. However, the high rates of reversals for some people remain problematic. Companson to a study 
using subjects' own HRTFs suggests that "listening through someone else’s ears" primanly results in an 
increase in front-to-back reversals. Note, though, that the existence of free-field confusions shows that 
these reversals occur in the real world and so are not strictly the result of the simulation. 




Several stimulus characteristics may help to minimize these errors and are being examined in current and 
future studies. For example, the addition of dynamic cues correlated with head-motion, well-controlled 
environmental cues generated from models of room acoustics, and correlated visual cues should improve 
the ability to resolve these ambiguities and substantially enhance the efficacy of any virtual acoustic 
display. 

System Architecture 

Displays: Stereo headphones. 

Input: Polhemus, 4 analog sound sources 

Rendering: Convolvotron 

Computation: 386 PC 

Software: Custom. 

^^uK? t^R. (1991). Challenges to the successful implementation of 3-D sound. Journal of the Audio 

Enaineerinq Society, 39(11), 864-870. . , 

Begault, D. R. (1992). Perceptual similarity of measured and synthetic HRTF filtered speech stimuli. 

Journal of the Acoustical Society of America, 92(4), 2334. , 

Begault, D. R., & Wenzel, E. M. (In Press). Headphone Localization of Speech. Human Factors 
Wenzel, E. M. (1992). Localization in virtual acoustic displays. Presence: Teleoperators and Virtual 

Wenze^^M^Wightm^L F°L, & Kistler, D. J. (1991). Localization of non-individualized virtual acoustic 
display cues { pp. 351 -359). New York: The Association for Computing Machinery. 
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Virtual Reality Applications in the Earth and Space Sciences 

Principal Investigators: Horace G. Mitchell and Daniel S. Spicer 
NASA Goddard Space Flight Center 

Research Objective 

To provide Earth and space science researchers with a VE based on a familiar user interface so as to 
introduce and evaluate VEs as a research tool. 


Research scientists working in the Earth and space sciences require VE tools that they can immediately 
utilize as an extension of their existing tools and methods. In this way they can minimize both learning and 
development time and evaluate VEs directly as a research tool. The Flow Analysis Software Toolkit 
(FAST) is a popular, NASA-developed visualization tool, specifically designed to visualize the physics of 
three dimensional flows and optimized for high-end graphics hardware. By utilizing a subset of F AST 
modules for data input, analysis, and visualization control, a VE version of FAST is being developed (VR- 
FAST) which will allow research scientists to work with a VE package that they are already familiar with. 

Initial development of the VR-FAST is currently in progress. The development hardware system has been 
purchased and delivered to Sterling Software, the developers of FAST. Helmet tracking modifications 
have been integrated and 75% of the glove software is done (innate gestures and learn gestures done, 
with applications integration in progress). 


Future Plans , 

The initial, deliverable VR-FAST system will be completed this year (FY93). Immediate subsequent 
development will focus first on a multiple user version, for collaborative research efforts, and then on 
versions that are distributed over multiple machines and graphics hardware and are parallelized for high 
performance. 


System Architecture 

Display: Dual display color helmet 

Input: VPL DataGlove, keyboard, mouse 

Rendering: Silicon Graphics Dual Reality Engine 

Computation: Silicon Graphics Dual Reality Engine 

Software: FAST 


Skywriter 

Skywriter 
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Workload Assessment Using a Synthetic Work Environment 


Principal Investigator: Susan Adam 
Co-investigators: Manny Diaz 
NASA-Johnson Space Center 


Research Objective 

To quantify the additional mental workload imposed by microgravity, 
workload imposed by proposed spacecraft systems. 


To develop tools for assessing the mental 


St at u s 

A series of mental workload studies using a synthetic work environment has been completed. These stud- 
ies sought to demonstrate the utility of Response Surface Methodology (RSM) central-composite designs 
for predicting mental workload. The intent was to conduct the first in a series of studies that would 
ultimately establish systems operating conditions for proposed spacecraft systems that do not overload 
the capabilities of the operator. Eight subjects participated in a testing situation comprised a synthetic 
work environment consisting of four tasks: visual monitoring, Sternberg memory, arithmetic computations 
and auditory monitoring. 

Preliminary results suggest that RSM provides an effective tool for evaluating operating conditions in syn- 
thetic work environments in terms of the imposed mental workload. Plans call for augmentation of the first- 
order model to a second-order central composite design to more accurately characterize the response 
surface. Then, the predictively validity of the response surfaces will be established, further demonstrating 
the utility of RSM for evaluating the operating conditions in proposed spacecraft systems. 


The results of these studies will advance current understanding of the relation between the simultaneous 
affects of various systems operating conditions and how they impact the mental workload imposed by 
spacecraft systems proposed for extended duration spaceflight. However, the synthetic work ... . 

environment does not provide the fidelity necessary for comprehensive human workload analysis Virtual 
reality technology can be expected to provide a new level of fidelity and an enhanced capability for in- 
depth evaluation of the workload imposed by microgravity and by proposed spacecraft systems. 


System Architecture 


Displays: 
Input: 
Rendering: 
Computation 
Software 


Compaq PC 

Mouse 

M68000 

M68020 

SYNWORK1 


NASAVE R&T 


67 


Device for Orientation and Motion Environments (DOME) - 
Preflight Adaptation Trainer (PAT) 

Principal Investigator: Deborah L. Harm 

Co-investigators: Donald E. Parker,. Miami University-OH, and Millard F. Reschke 
NASA Johnson Space Center 

Research Objective 

To develop training devices and procedures to preadapt astronauts to the sensory stimulus 
rearrangements of microgravity. The trainers are intended to demonstrate sensoiy phenomena likely to be 
experienced inflight and immediately postflight, allow astronauts to train preflight in an altered sensory 
environment, alter sensorimotor reflexes (appropriate for microgravity), and eliminate or reduce space 
motion sickness and orientation and motion disturbances. 


The DOME-PAT system is currently being used to train astronauts in the recognition of, and quantitative 
description of perceptual experiences associated with head and body movements made on-orbit, during 
entry and immediately postflight. Postflight, astronauts report which set(s) of conditions in the DOME are 
similar to their flight related perceptual experiences and help evaluate potentially useful training tasks. The 
system is a 3.7 m diameter spherical dome with a 1 .8 m diameter hole in the bottom. The inner surface of 
the dome serves as a projection surface for two Triuniplex video projectors with custom wide angle optics 
providing a 100° X 170° field of view. An adjustable trainee restraint assembly, along with the projectors, is 
mounted on a 1 .8 m diameter rotating base which fills the hole in the bottom of the dome. The trainee 
restraint adjusts for positioning the trainee to: (f) sit upright, (2) lie on either the left or right side, or (3) lie 
supine. A 6 DOF isometric joystick or forceplate is used by the trainee and/or the instructor for virtual 
motion within the visual environment. The joystick can be used to control real whole body rotation when 
the rotating base is enabled. Position signals derived from torque sensors in a trainee head restraint 
assembly can also be used to drive the visual scene in a manner appropriate for either real or intended 
head movement. Real head movements are permitted only in a plane orthogonal to gravity to eliminate a 
gravity stimulus to the gravity receptors in the inner ear. 


Future Plans , , , , J L 

In the near-term future: (f ) new training tasks/protocols will be developed and evaluated by astronauts as 
part of an ongoing detailed supplementary objective (DSO) activity, (2) criteria for evaluating the efficacy of 
different training protocols will be determined, and (3) a visual data base for simulating EVA will be devel- 
oped. Ultimately, we expect the full complement of training procedures to be implemented as part of the 
astronaut's operational training. For long-duration missions (STS and Space Station) and Mars missions 
we expect to develop a compact inflight VR system and protocols for maintaining adaptation to Og, 1/3g 
and 1g. 


System Architecture 

Displays: Space Lab, Middeck/Flight deck, and polarized checkerboard room 

Input: 6 DOF isometric force joystick, torque sensors in trainee head restraint system 

Rendering: Silicon Graphics 4D-440 VGX, two Triuniplex projectors with custom optics 

Computation: Silicon Graphics 4D-440, Vision Works (Paradigm) 

Software: In-house 


1. Reschke, M.F., Parker, D.E., Harm, D.L..& Michaud, L: (1988). Ground-based training for the stimulus 

rearrangement encountered during space flight. Acta Otolaryng (Stock), Suppl.,60, 87-93. 

2. Harm, D.L., & Parker, D.E. (1989). Mode A Preflight Adaptation Trainer. In NASA Technical 

Memorandum 1 00 473. Houston, TX: JSC. ..... , „ 

3. Parker D.E., Parker K.L. (1990). Adaptation to the simulated rearrangement of weightlessness, in g.h. 

Crampton (Ed.), Motion and Space Sickness (pp. 247-262). Boca Raton: CRC Press. 
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Training for Eva Satellite Grapple 

Principal Investigator: R. Bowen Loftin (NASA JSC and University of Houston) ..... 

Additional Investigators: Lui Wang (NASA/JSC); Mar* Voss and Jeff Hoblitt (LinCom); Lac Nguyen (CSC) 
NASA-Johnson Space Center 

To develop a^proofof concept training environment that provides astronauts with a simulation of satellite 
dynamics that match those observed during STS-49. 

St atu s 

A proof of concept VE for EVA satellite grapple is under development. Elements include models of the 
orbiter, the RMS, and Intellsat. The Intellsat model is dynamic and will respond to impulses .imparted I by _a 
human hand or a hardware fixture. A Polhemus device is used to track hand or fixture motion. The mode 
detects collisions and infers the impulse imparted to the payload from a simple mass model of the hand or 
fixture and its velocity on collision with the payload. 

If *the r proof 'of 1 concept system is judged to be valuable for training by training personnel and experienced 
astronauts, a more complete model will be developed and delivered as a training system for future payload 
retrieval missions. 


System Architecture 

Displays: VPL EyePhone, LX and HRX 

Input: VPL DataGlove, Model 2; Polhemus 

Rendering: SGI 320VGX 

Computation: SGI 320 VGX 

Software: TDM, OOM (in-house) 
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Shared Virtual Environments 


Principal Investigators: R. Bowen Loftin (NASA JSC and University of Houston) and Joe Hale (MSFC) 
Additional Investigators: Lui Wang (NASA/JSC); Lac Nguyen (CSC); Mark Voss (LinCom) 
NASA-Johnson Space Center 

Research Objective 

To develop the capability for sharing VEs vis long-distance networks. 


The capability of sharing the same VE between JSC and MSFC has been demonstrated. Such a shared 
environment permits personnel at both centers to simultaneously observe and interact with the same 
virtual objects. The use of existing networks imposes unpredictable latencies due to other network traffic. 


The installation of a dedicated communication link between JSC and MSFC is planned. The nature of the 
shared environments will be increased in complexity and more interactive tasks will be performed. 


System Architecture 

Displays: VPL EyePhone, LX and HRX 

Input: VPL DataGlove, Model 2; Polhemus 

Rendering: SGI320VGX 

Computation: SGI320VGX 

Software: TDM, OOM (in-house) 
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Space Station Cupola Training 

Principal Investigators: R. Bowen Loftin (NASA JSC and University of Houston) and Beth Holewinski 
(NASA/JSC) 

Additional Investigators: Lui Wang (NASA/JSC); Lac Nguyen (CSC); Mark Voss (LinCom) 
NASA-Johnson Space Center 

Research Objective _ . . 

To develop a training environment that supports astronaut training for Space Station cupola operations. 


A VE for Space Station cupola operations training is under development. Elements include models of the 
exterior of Space Station and the interior of the cupola. Scenarios to be supported include remote 
operation of the Space Station RMS with payload grapple and interaction of EVA personnel with Space 
Station RMS. 


Future Plans 

Transition of models from VPL software to in-house software; testing the efficacy of this approach to 
cupola training compared to dome and/or pancake window simulation. 


Architecture 

VPL EyePhone, 
VPL DataGlove, 


System 

Displays: 
Input: 
Rendering: 
Computation: 
Software: 


SGI 320 VGX 
SGI 320 VGX 
VPL Swivel 3-D, 


LX and HRX 
Model 2; Polhemus 


Body Electric, Isaac; TDM, OOM (in-house) 


NASA VE R&T 


71 


Virtual Science Laboratories 


Principal Investigator: R. Bowen Loftin 
NASA-Johnson Space Center and University of Houston 

Research Objective 

To develop virtual laboratories that provide students access to experiments that cannot be performed in 
"real" laboratories, thereby enhancing student mastery of difficult concepts and increasing student 
motivation for science. 


A prototypical Virtual Physics Laboratory has been constructed. This laboratory provides for user control 
of gravity (both magnitude and direction), friction, atmospheric drag, and the coefficient of restitution for 
elastic bodies. Measurements of both time and length can be performed. Trajectories of objects can be 
traced for future measurement. Time in this "world" can be stopped and started at will to support both 
observation and measurement. Objects such as a plane pendulum (with variable length), balls, and plane 
surfaces are available for experimentation. 


Future Plans J . . .... .... . 

During the fall, 1992, formative testing of the Virtual Physics Laboratory will be conducted with both high 
school and college students. Based on this evaluation, a second version of the laboratory will be 
developed using in-house software. Additional plans call for the development of virtual laboratories to 
support chemistry, biology, and earth science. 


System Architecture 

Displays: VPL EyePhone, LX and HRX 

Input: VPL DataGlove, Model 2; Polhemus 

Rendering: SGI 320VGX 

Computation: SGI 320 VGX 

Software: VPL Swivel 3-D, Body Electric, Isaac 


References 

1. Loftin, R. B., Engleberg, M., and Benadetti, R. (1992, 
presented at CAI, Washington, DC. 


October). A Virtual Physics Laboratory. Paper 
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Realistic Lighting Models for Virtual Reality 

Principal Investigator: James Maida (NASA JSC) per* 

Additional Investigators: Ann Aldridge, Abh.lash Pandya ( Lockheed ESC) 

NASA Johnson Space Center 

To develop a^ornprehenslve lighting model to simulate realistic liohling conations ter potential use in 
virtual reality environments. The research will include study of camera and lens behavior, material 
properties, and light parameters. 

Computer simulation of lighting scenarios is important. Typical viewing simulations are often simple color 

nuu«W and «Mbig 

Drooerties are valid. These are the areas of research currently underway. . , 

i Resources of NASA's lighting laboratory are being used for the collection of lighting and mater a 
JelKt? SJtp^TW? data°wi( be usid to correlate input parameters ot the light models to the 

Z^M^I^tensionsyre^deway whtarwnore accurately represent the relationships ot light attenuation 

3 ' 'Human' wrtSS StSs in light intensity. They modify the amount ot light 
LeSo me visual sysTem by automatically adjusting the aperture of the lens with an ins. This behavior 
sianificantlv chanqes what is visible and must be modeled. We are studying the image processing 

done by camera systems such as the human eye and shuttle 

cameras. 

S developed optimize a lighting system which incorporates realistic material and light parameters which 
might be used in a VE. 

System Architecture 

Displays: Silicon Graphics 360, VGX 

Input: Polhemus magnetic tracker 

Rendering: Silicon Graphics (R3000- RISC) 

Computation: Silicon Graphics (R3000- RISC) 

Software: In-house 

? Xiao e S C He, Torrance, K.,Sillion, F„ & Greenberg D. (1991). A 5?^ M ° de ‘ f ° f 

Light Reflection, Computer Graphics, 25(4), 175-186 (Proceedings SIGGRAP ). 

2. Greenberg, D. (1991). Light Reflection Models for Computer Graphics, Science, 244, 6 * 

3! Hall, R. (1988). Illumination and Color in Computer Generated Imagery (pp. 1 )• 

4. Folet P ?Van V Dam 9 A., Feiner, S, & Hughes, J. (1990). Computer Graphics - Principles and Practice 

5 Sillion n F E & Puech 2 C 8 n Ge^erafTwo-Pass Method Integrating Specular and Diffuse 
5 - S,I R 23. NO. 3. 335-344 (Proceedings SIGGRAPH '89). 
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Improving Human Model Reach for Virtual Reality 

Applications 


Principal Investigator: James Maida (NASA JSC) 

Additional Investigators: Ann Aldridge, Abhilash Pandya ( Lockheed ESC) 

NASA Johnson Space Center 

Research Objective 

To develop an accurate human computer reach model able to simulate complex three dimensional tasks 
with potential application in virtual reality applications. 

Status 

The most kinematically complex interaction to model in the human arm is the shoulder girdle. The motion 
of the shoulder complex involves several joints moving simultaneously with a complicated and dependent 
interaction. This shoulder motion was measured using a magnetic tracking device in all rotational planes. 
These measurements were used to validate and refine a clavicle /shoulder kinematic model. With this 
shoulder model, accurate reach positions can be predicted in all planes of motions to within, on average, 1 
cm of the measured data. 

Future Plans 

To extend the human reach model to include realistic and detailed motions of the hand and the spinal 
cord. 


System Architecture 

Displays: Silicon Graphics 360, VGX 

Input: Polhemus magnetic tracker 

Rendering: Silicon Graphics (R3000- RISC) 

Computation: Silicon Graphics (R3000- RISC) 

Software: In-house 


References 

I.Otani, E. (1989). Software Tools for Dynamic and Kinematic Modeling of Human Motion. Unpublished 

Master Thesis, University of Pennsylvania. 

2. Inman, V., Saunders, M. and Abbott L. (1944). Observations of the Function of the Shoulder Joint, 

The Journal of Bone and Joint Surgery, 26 (1), 1-30. 

3. Engin, A. and Turner, S.t (1989). Three-Dimensionaf Kinematic Modelling of the Human Shoulder 

Complex-Part I: Physical Model and Determination of Joint Sinus Cones. Journal of Biomechanical 
Engineering, 111, 107-112. 

4. Aldridge, A., Pandya, A., and Maida, J. (1992). Validation of the Clavicle/Shoulder Kinematics of a 

Human Computer Reach Model. Manuscript submitted for publication. 
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Human Computer Interface Research for Graphical 

Information Systems 

Principal Investigator*. Kevin O'Brien PHD. (Lockheed) « 

Additional Investigators: Benjamin Beberness (Lockheed), Steve Chrisman (Lockheed) 

NASA-Johnson Space Center 

ReSe The h Hum b a i n-Com e puter Interaction Laboratory (HCIL) has conducted an assessment tof 
information technology projected for lunar and Mars missions and the applicability of projected technology 
for supporting space exploration tasks (Advanced Information Technology Data-base). The task of 
selectinq the geographic location of planetary activities was identified as benefiting from technological 
advances Site selection is important to a variety of larger tasks including conducting remote surveys, 
^Standing and “Saco Activities such as scientific exploration. Of specific interest to those .nvdved 
in site selection is the rapidly developing Geographic Information Systems (GIS) technology. While the 
Kit of GIS's to site selection in earth based activities has been demonstrated it « of fields 
(urban planning, petroleum exploration and drilling, land management), many challenges to t e 
development of the technology exist, and the development of the technology for the purpose of 
supporting planetary exploration is as yet unexplored. The HCIL has completed a background report on 
&5S invoked in the application of GIS technology to space exploration, including reviews of four 
existing personal-computer based GIS's. 

Our current focus is defining, evaluating, and illustrating user interface issues relevant to a GIS s u P 'P® h in ^ 
S S dedsion making. CuSently we are collaborating with Space Shuttle Earth Observation 
P^oted^SSEOP) 1 people in developing a GIS to aid the Flight Crew Office in performing site selection task. 

us to: 1) obtain a good understanding how a GIS can be built from he ground up 
iT “%££ Sed environment, and 2) research issues are expected to evolve that we will test a t a later 

date. 

Lona term aoals are' 1) demonstration package on GIS user interface issues, 2) Completion of Issues in 
th£ E 3 Geoaraphto Information Systems", 3) empirically examine GIS human-computer interface 
fedS site selection tasks, 4) identify factors affecting user performance in a 

planetary GIS work task, and 5) develop parameter models of site selection for earth based tasks, p 
planetary site selection, and projected planetary site selection. 
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Macro-Ergonomics and Scaleable User Anthropometry 

Principal Investigator: Joseph P. Hale 

Additional Investigators: Michael Flora, Peter Wang (NTI) 

NASA Marshall Space Flight Center 


To develop, assess, and validate VR as a macro-ergonomics analysis tool for considering operational, 
viewing and reach envelope requirements in the topological design of work areas. To develop, assess, 
and validate scaleable user anthropometry attributes. A study will compare Virtual Payload Operations 
Control Centers (VPOCC) with the existing Payload Operations Control Center (POCC). 


Two VPOCCs have been developed that contain the basic objects of the POCC (e.g., tables, monitors, 
printers, communication panels, etc.) and their spatial layout. One "operational VPOCC permits the 
relocation of objects that are generally moveable and moved in an operational environment (e.g., 
keyboards). Another "non-operational" VPOCC permits the relocation of ail objects that can be moved in 
the "real world" (e.g., tables). 

Test scenarios will be performed in both the POCC and operational VPOCC and their results compared to 
ascertain what, if any, distortions arise in a Virtual World (VW). The test scenarios will focus on what one 
can see from a variety of eye reference points using a range of real and virtual anthropometnc sizes. These 
scenarios will also include operationally-driven components such as translation paths among the various 
console-, printer-, fax-, file-locations. 

An algorithm has been developed to rescale user anthropometric attributes to any desired virtual anthro- 
pometry. Thus, a 95th %-iie male could view and reach as a virtual 5th %-ile female and vice-versa. 

The non-operational VPOCC will be used to explore alternative POCC configurations, i.e., different 
topological arrangements of consoles, printers, faxs, files, etc. The various configurations will be 
compared using much the same methodology developed for the operational VPOCC. 


FuturG plans 

As confidence is gained in VR as a macro-ergonomic analytical tool, it will be applied in upcoming 
topological design efforts such as the Space Station Payload Operations Integration Center (POIC). 
Specific options to be addressed in the design of the POIC, in addition to "standard topological design 
issues include slanted walls (to cover utility runs) and tiering of the control room and viewing area floors. 
The validated scaleable user anthropometry capability will be applied in future macro- and micro-ergonomic 

analyses. 


System Architecture 
Displays: VPL Eyephones 

Input: VPL DatagGlove model 2, Polhemous, keyboard, mouse. 

Rendering: Silicon Graphics 310 VGX, Silicon Graphics 320 VGXB 

Computation: Macintosh II fx 
Software: Swivel 3D, Body Electric, ISAAC 
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Micro-Ergonomics: Virtual and Fomecor Mock-Ups 


Principal Investigator: Joseph P. Hale 

Additional Investigators: Michael Flora and Peter Wang (NTI) 

NASA Marshall Space Flight Center 


To develop, assess, and validate VR as a micro-ergonomics analysis tool for considering operational, view- 
ing and reach envelope requirements in the spatial layout of workstations and worksites. To develop, as- 
sess, and validate scaleable user anthropometry attributes. A study will compare a Virtual Crew Interface 
Coordinator (VCIC) console with a proposed redesigned Crew Interface Coordinator (CIC) console. 


The CIC console is part of the Payload Operations Control Center (POCC). The CIC position is analogous 
to the Mission Control Center's “Capcom" position, communicating directly with the Spacelab payload 
crew about payload operations issues. Two types of virtual mock-ups will be developed. One operational 
VCIC console will permit the relocation of objects that are generally moveable and moved in an operational 
environment (e.g., keyboards). Another "non-operational" VCIC console will permit the relocation of all 
objects that can be moved in the "real world" (e.g., monitors, worksurfaces). 


Test scenarios will be performed on both a "Fomecor" mock-up and the VCIC console and their results 
compared to ascertain what, if any, distortions arise in a Virtual World (VW). The test scenarios will focus on 
what one can see from a variety of eye reference points and on what one can touch from a variety of 
shoulder reference points using a range of real and virtual anthropometric sizes. An algorithm has been 
developed to rescale user anthropometric attributes to any desired virtual anthropometry. Thus, a 95th /o- 
ile male could view and reach as a virtual 5th %-ile female and vice-versa. Results of these analyses will also 
be compared to determine the relative merits of VR vis-a-vis an existing, "standard Human Factor's tool 
(i.e., "Fomecor" mock-up). 


The non-operational VCIC console will be used to make real-time design changes with immediate Human 
Factors analyses of the consequences. This capability is one of the benefits of VR analyses. Because the 
VWs are nothing more than computer files, design changes can be done more quickly and more 
candidate configurations can be subsequently analyzed than is currently possible with existing, 
"standard" Human Factor tools. The Fomecor mock-up will then be updated, based on the design of the 
"refined" virtual mock-up, and a second validation study with the CIC console will be conducted. 


As conlidencefs gained in VR as a micro-ergonomic analytical tool, it will be applied in future workstation 
and worksite design efforts as might be found in the Space Station Payload Operations Integration Center 
(POIC). The validated scaleable user anthropometry capability will be applied in future macro- and micro- 
ergonomic analyses. 


System Architecture 
Displays: VPL Eyephones 

Input- VPL DatagGlove model 2, Polhemous, keyboard, mouse. 

Rendering: Silicon Graphics 310 VGX, Silicon Graphics 320 VGXB 

Computation: Macintosh II fx 
Software: Swivel 3D, Body Electric, ISAAC 


NASA VE R&T 


77 


Microgravity Mobility and Ergonomics 

Principal Investigator: Joseph P. Hale 

Additional Investigators: Michael Flora and Peter Wang (NTI) 

NASA Marshall Space Flight Center 

Objective 

To develop, assess, and validate VR as a Human Factors design analysis tool for considering operational, 
viewing and reach envelope requirements in a microgravity environment. To develop, assess, and validate 
techniques and methods that provide some of the various advantages and disadvantages of reaching and 
maneuvering in microgravity. To compare the results from a "virtual analysis" with a previously conducted 
analysis relating to the operation of the Electromagnetic Processing Facility (TEMPUS), an experiment to 
fly on the Second International Microgravity Laboratory (IML-2) Spacelab mission. 

Status 

TEMPUS is a levitation melting facility for processing of metallic samples in an ultra-clean environment. 
Sample positioning and heating can be controlled separately by two independent RF oscillating 
circuits.The issue driving the previously conducted analysis was whether a crewmember could adequatefy 
control the position of a sample in the facility with controls located in the right half of Rack 10 while 
monitoring the results on a CRT in the right half of Rack 8. The CRT was co-planar with the rack face and 42 
inches away from the controls. A full-scale part-task Fomecor mock-up of both racks was fabricated to 
determine the crewmembers ability to view the CRT while touching the controls. 

A virtual mock-up of racks 8 and 10 has been developed and placed inside of a virtual Spacelab module. A 
method to enable the various advantages and disadvantages of reaching and maneuvering in a 
microgravity environment has been developed, within existing VR technology capabilities and limitations. 
In particular, the user can manipulate the attitude of the Spacelab, as a whole, while "grabbing" a handrail, 
giving the egocentric perception of microgravity mobility. Viewing and reach envelope analyses will be 
conducted and the results compared with the previously conducted "Fomecor study". This study will also 
include scaleable user anthropometry, currently being developed and assessed in another study. This will 
help to further refine this capability. 

Future Plans 

In addition to applying the VR analytical tools as they "come on-line", the VR toolkit will be further refined 
and augmented. One particularly valuable enhancement to be addressed in this area is the incorporation 
of an anthropometric model. This would enable dynamic work envelope analyses. The anthropometric 
model should include link lengths to reflect a broad anthropometric design range (e.g., 5th %-ile female 
through the 95th %-tile male) and realistic joint range-of-motion capabilities and constraints. A candidate 
subject to be used to evaluate and refine this feature is an unplanned In-Flight Maintenance (IFM) that 
occurred on Spacelab 3 (Drop Dynamics Module (DDM)). The goal would be to actually recreate the IFM 
environment and operation, then compare this virtual IFM experience with the actual flight experience. 
This would include reference to video and audio recordings of the on-board operation, written logs, and 
participation of the actual Spacelab crew involved in the IFM operation. 

System Architecture 
Displays: VPL Eyephones 

Input: VPL DatagGlove model 2, Polhemous, keyboard, mouse. 

Rendering: Silicon Graphics 310 VGX, Silicon Graphics 320 VGXB 

Computation: Macintosh II fx 

Software: Swivel 3D, Body Electric, ISAAC 



Chapter 4: Near Term Mission Support 


This chapter contains examples of possible applications of VE technology to near term NASA Space 
missions. The chapter is organized by missions or sets of missions. The list is by no means exhautive. 


Chapter Organization 


Aeronautics r . 

Head-Mounted Displays for Aircraft Assembly 
Safer Communications in Air Traffic Control 
Aeronautical Virtual Acoustic Displays . _ , 

Space Transportation System, Space Station, Exploration 
Automated Training Evaluation and Improvement 
Designing Tools for Humans in Space 
Exploration (Lunar/Mars) and Planetary Science 
In Situ Training 
Exploration (Lunar/Mars) 

Crew Health and Performance 

Dynamic Virtual Environment Database 

Task Analysis 

Crew Health — Medical 

Crew Health - Entertainment 

Crew Health -- Virtual Confidant 

In Situ Training 


Planetary Science . 

Shared Experience: Science, Operations, and Education 

Proficiency Training 
Space Transportation System 

After-the-Fact Analysis, Accident or Event Reconstruction 

Hubble Space Telescope Maintenance/Repair 

EVA/RMS Training and Procedures Development for HST Repair 

Crew Training for Satellite Retrieval and/or Repair 

EVA Operations Development 

RMS Training 

Near-Term VR Applications in Spacelab 
Space Transport System and Space Station 
Crew Health and Performance 
SAFER Engineering Test and Development 

Space Station 

Manipulator Systems Training 
Space Station Construction 
In Situ Training 

Crew Medical Restraint System 
Space Station Operations (IVA and EVA) 

Near-Term VR Applications in Space Station 
The Great Observatory Series 

Near-Term VR Applications in the Design of the Advanced X-ray 

Astrophysics Facility 
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Head-Mounted Displays for Aircraft Assembly 

Mission: Improve productivity in aircraft manufacturing 
Center: Ames Research Center 

Problem* Assembly of commercial aircraft requires the transfer of spatial information concerning 
wirinq rivets, and hydraulic lines from CADCAM databases onto the skin of aircraft subsections 
under’construction. Some of this information is used automatically during assembly but much is 
currently used semiautomatically by skilled technicians who need to visualize the spatial layout of 
the design elements on or in their work surface. Current manufacturing techniques require them 
to move back and forth between displays or blueprints to as they transfer templates of layouts to 
complete fabrication of aircraft subsections in what is currently a labor-intensive, inefficient method 
of aircraft manufacturing. 

VR Appllcatlon/Approach: A head-mounted, see-through stereoscopic display presented as 
a spatially conformal projection of the layout information visually superimposed on the work surface 
will provide a convenient and more efficient means for the aircraft worker to transfer the necessary 
information. The visual and oculomotor characteristics of such displays, however, must be carefully 
designed so that the stereoscopic virtual images that they present are seen in the correction 
direction and at the appropriate apparent distance. 

Perceptual errors in distance with such displays are especially troublesome if the viewer attempts 
to manipulatively interact with the virtual images treated as virtual objects. Recent observations of 
the apparent egocentric distances of stereoscopic virtual images have indicated that the perceived 
distance of virtual stereo images can be very labile, depending upon individual differences and the 
physical background against which they are seen (Ellis & Bucher, 1992). The interaction can be 
especially striking when a extended physical object is introduced as the apparent distance of the 
virtual object. Though in this condition vergence and accommodation conflict may be expected to 
be small, the apparent distance to the virtual image is for most observers shifted towards the viewer 
even when the distance based on disparity is within the approximately 1 meter setting of resting ^ 
accommodation and vergence. The presumptive cause of this reduction is that the optical « overlay 
of the virtual image, makes it appear to occlude the background, and thus appear closer P anned 
research will test this hypothesis by studying the effects of varying the capacity of the virtual image 
to occlude the background and assist development of compensatory techniques to correctly 
display the apparent distance to spatial conformal, virtual images, close to the human operator. 

Benefits: Successful design of head-mounted see-through stereo displays for fabrication will 
markedly increase assembly worker productivity and will provide information for improved design of 
nonsee-through head-mounted stereo viewing systems which may be used for visualizing and 
programming industrial robots as well as telerobots and for human operators procedure training. 

Coordination: Aerospace Human Factors Research Division 


Prepared by: 

Stephen R. Ellis 
NASA ARC 
415 604 6147 
silly@eos.arc.nasa.gov 
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Safer Communications in Air Traffic Control 


Mission: Terminal Area Productivity Program, AS/A 
Center: Ames Research Center 

Problem: In order to facilitate the increased traffic in the terminal area mandated by TAP, there 
must be a plan to facilitate an increased number of radio communications between tower and 
incoming and outgoing aircraft. Currently, single earpiece systems are used for one incoming 
channel, with additional communications relegated to a loudspeaker or telephone handset. There 
will also be a need for an increase in the number of warnings and alerts, within a high-stress 
environment that already places maximal demands on the visual system. 


VR Appl leaf lon/Approach : The approach involves implementing a virtual auditory display for 
each controller's workstation, with air traffic controller s using 2-ear headsets. The cognitive 
workload involved in separating various communication streams can be reduced by using 3-D 
sound techniques within the auditory display to assign communications to specific locations. The 
placement of these communications can correspond to proximity, flight phase, or actual location 
out the window (with the addition of head-tracker technology). In addition to radio 
communications, psychoacoustically optimized aural alerts are desirable for integrating within the 
auditory display because of the increased load on the visual system for monitoring situational 
awareness These aural alerts can be used either in addition to or as a substitute for visual alerts; 
their placement in virtual auditory space can be separated from radio communications, and 
spatialized according to operational criteria. Active noise reduction techniques allow the use of 
lightweight headsets and a reduction in background noise. 


Benefits: Initial investigations have shown about a 6-7 dB improvement in intelligibility using a 4- 
channel system A lessening of the demands on the visual modality will be beneficial. Because an 
auditory display controls all of the sound input to a controller's auditory system, distracting noise 
sources can be significantly attenuated, and a single control device can display all voice and alert 
communications in a coherent manner, either manually, or automatically. The load on the visual 
modality can be decreased. 


Coordination: Aerospace Human Factors Research Division 


Prepared by: 

Elizabeth Wenzel and Durand Begault 
NASA ARC 

415 604 6290/ 415 604 3920 

beth@aurora.arc.nasa.gov 

db@eos.arc.nasa.gov 
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Aeronautical Virtual Acoustic Displays 

Mission: Terminal Area Productivity Program, AS/A 
Center: Ames Research Center 

Problem: Current "glass cockpit" human factors design has relegated considerable attention to 
improving visual displays. Contrasting this, the display of acoustic information does not reflect an 
integrated design philosophy. The result is that desired signals- radio communications, inter- 
aircraft communications, auditory warning signals & alerts, feedback from aircraft engine system, 
control panel switch activation, etc.- are heard from indiscriminate sources (headset, various 
speakers) against a high level of background noise (76 dB at the left ear of the pilot in a Boeing 
737). The priority of acoustic information under the best circumstances is a function of loudness 
(e.g., a loud fire alarm bell), an undesirable state of affairs in a high-stress human interface. 

VR Applicatlon/Approach: The approach involves implementing a virtual auditory display, with 
pilots using 2-ear headsets. The placement and intensity of all of the auditory input to the pilot is 
strategically placed in a concordant manner that allows maximal intelligibility, prioritization, and 
positive redundancy. The cognitive workload and overall fatigue levels involved in separating 
various communication streams over one ear can be reduced by using 3-D sound techniques to 
separate auditory input to different locations. Previous studies at NASA Ames have shown a 6-7 
dB improvement in intelligibility with 3-D auditory displays, which would help minimize hearback- 
readback problems between pilot and ATC; especially for airlines with hub operations, where call 
signs across one frequency are very similar. In addition to radio communications, 
psychoacoustically optimized aural alerts are desirable for integrating within the auditory display 
because of the increased load on the visual system for monitoring situational awareness. These 
aural alerts can be used either in addition to or as a substitute for visual alerts; their placement in 
virtual auditory space can be separated from radio communications, and spatialized according to 
operational criteria. The 3-D sound TCAS studies conducted at Ames have shown about a .5 
second improvement in target acquisition time when using only 3-D sound alerts for aurally guided 
search, compared head down, standard TCAS displays; and a 2.5 second improvement when 
comparing one ear to two ear 3-D audio alerts. The placement of aircraft system alerts should 
correspond to meaningful, positive-redundant locations: e.g., the "left engine fire" verbal 
announcement comes from the left side. Aural feedback from non-visible sources (e.g., landing 
gear, control panel switches above the head) or visible "virtual" sources (e.g., the touch screen 
switches on the CDU panel) can be readily synthesized to give aural feedback about positive 
engagement, allowing the eyes to remain out-the-window. Active noise reduction techniques 
allow the use of lightweight headsets and a reduction in background noise, minimizing fatigue and 
the need to turn the head to direct the voice, e.g., pilot to flight engineer communications in a 
Boeing 727. Active noise reduction also allows reducing the level of one’s own voice against noisy 
backgrounds (the Lombard effect), also contributing to less fatigue. 

Benefits: Allowing pilots to have a quieter, organized presentation of desirable auditory 
information will allow safer operation of aircraft. This is certain to occur due to the known benefits 
from reduction of noise levels; the increase in intelligibility; positive redundancy between visual 
and auditory modalities; aural feedback from non-visible sources; and the advantages of aurally 
guided visual search as demonstrated in the NASA Ames 3-D audio TCAS studies. 

Coordination: Aerospace Human Factors Research Division 

Prepared by: 

Elizabeth Wenzel/ and Durand Begault 
NASA ARC 

415 604 6290/ 415 604 3920 
beth@aurora.arc.nasa.gov; db@eos.arc.nasa.gov 
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Automated Training Evaluation and Improvement 


Mission: Space Transportation System, Space Station, Exploration (Lunar/Mars) 

Center: Johnson Space Center 

Problem: Systematic and objective collection of crew performance data during training for 
evaluation and for direct input into improvement measures Is very difficult under present 
conditions. Data is now based on subjective crew comments and the opinion of trainers who must 
concentrate on conducting training, not evaluating it. 


VR Appllcatlon/Approach: When developing a virtual reality (VR), one needs to incoiporate a 
large number of scenarios, each built up of a combination of specific events. If one has a list of 
these events and their parameters ahead of time, one can record how often an astronaut in the VR 
chooses to make them occur. In essence, you record a person’s behavior, or at least the 
consequences of their behavior. The VR computer can also record instances where an astronaut 
wishes to do something in the VR but cannot. These occurrences can be used to augment the VR 
capability and repertoire, or they can be recorded as errors and trained away with feedback. Other 
astronauts could then actively or passively experience such errors to recognize and avoid them. 
Data could even be collected and applied in-flight during missions to the moon and Mars on the 
same tasks completed on the ground-something which is not possible now because of obvious 
constraints. 


Benefits: Automated astronaut performance data collection during training would ensure 
objective data and continuous improvement. It would save time on the ground and in-flight, and 
would allow less expensive program upgrades. Operational experience would also be preserved 
from one crew (and astronaut) to another. 

Coordination: SP/Man-Systems Division, CB/Astronaut Office, DG/Training Division, DT/Space 
Station Training Division 


Prepared by: 

Thomas F. Callaghan/C95 
Lockheed ESCO 
2400 NASA Road One 
Houston, TX 77058 
(713) 333-7820 
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Designing Tools for Humans in Space 

Mission: Space Transportation System, Space Station, Exploration (Lunar/Mars) 

Center: Johnson Space Center 

Problem: Tools used in the exploration of space are often specially designed and manufactured 
in small numbers. The cost of designing and testing these tools is high because it cannot be offset 
with the reduced cost per unit gained from mass production. How can the cost of designing and 
testing these tools be reduced without compromising quality? 

VR Application/Approach: If the design phase of a tool can incorporate a simulated utility 
phase, the iterative process of design and redesign will converge to a solution faster. Using VR as 
a means to simulate tool utilization will permit the designer to manipulate a tool with virtual hands 
and feet to test its performance in an infinite variety of scenarios while designing the tool. Issues 
such as clearance, leverage, ease of use and constraints can be analyzed in the design cycle. 
While not a substitute for the actual testing a manufactured tool, resolution of many of these issues 
will increase the probability of a successful design. 

Benefits: The closer a tool design is to a quality product prior to manufacture the better and less 
costly the final product will be. Use of VR to enhance tool design will 1) permit redesign prior to 
manufacture to reduce cost and 2) increase the level of testing at the redesign stage to improve 
quality. 

Coordination: SP/Man-Systems Division 

Prepared by: 

James Maida/SP34 
NASA JSC 
713 483 1113 
maida@graf6.jsc.nasa.gov 
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In Situ Training 

Mission: Exploration (Lunar/Mars) and Planetary Science 
Center: Johnson Space Center 

Problem: Long-duration missions, such as those envisioned in the SET, will require refresher 
crew training for infrequently performed tasks. Simulators containing hardware elements cannot 
be flown to support such training. 

Description: Virtual environment technology would provide access to a complete simulation for 
crew training while in transit. Such simulation could encompass both EVA and IVA tasks and would 
be especially effective for infrequently performed tasks. This training could be extended to 
include planetary surface excursions and activities. 


Benefits: Provide training capabilities not available through other mechanisms; enhance training 
and probability of mission success. 

Coordination: 

CB/Astronaut Office 
DG/Training Division 
DT/Space Station Training Division 
ER/ Automation & Robotics Division 
PT4/Software Technology Branch 
SP/Man-Systems Division 

Prepared by: 

R. Bowen Loftin/Mail Code PT4 
NASA/Johnson Space Center 
Houston, TX 77058 
713-483-8070 (voice) 

713-244-5698 

bloftin@gothamcity.jsc.nasa.gov 
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Crew Health and Performance 

Missions: Exploration (Lunar/Mars) 

Center: Johnson Space Center 

Problem: Lunar and Mars missions will require astronauts to develop and maintain appropriate 
neurosensory and sensory-motor responses to three different gravito-inertial environments (1g, 
Og, and 1/6 or 1/3 g). The adaptive responses developed for one environment will not be 
appropriate for the other two gravito-intertial environments. Generally speaking, the longer the 
exposure duration to a given gravity environment the more complete the adaptation to that 
environment which will likely result in a longer period of re-adaptation to either of the other gravity 
environments. Transition periods between different gravito-inertial environments can result in 
postural, gait, and visual instabilities, disturbances in eye-hand coordination, and motion sickness 
symptoms: all of these may impact crew health and performance. 

VR Application/Approach: A head-mounted system, configured with head position or rate 
sensors, could be used to present and desired visual environment; a dataglove and or joystick 
could be used for performance of predetermined tasks. The head and limbs of the user could be 
loaded to simulate the appropriate gravitational force. In addition, a VR system integrated with an 
on-board centrifuge may be used to generate different gravitational force environments. The 
centrifuge could take the form of a bicycle mounted on a circular track inside the payload bay; the 
force field would be crew member generated, thus eliminating the need for Shuttle power to 
operate a traditional centrifuge. A set of tasks requiring eye-head and eye-hand coordination, and 
possibly locomotion could be designed with visual and tactile or force feedback correct for specific 
gravito-inertial environments. Crew members would practice various task scenarios throughout 
their mission. Such a system may include hardware for recording eye and limb movements. 

Benefit: Inflight VR systems could help crew members maintain appropriate neurosensory and 
sensory-motor responses for multiple gravito-intertial environments. 

Coordination: SD/Medical Sciences Division 

Prepared by: 

Dr. D. Harm/SD51 1 
NASA JSC 
713 483 7222 
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Dynamic Virtual Environment Database 

Mission: Exploration (Lunar/Mars) 

Center: Johnson Space Center 

Problem: The Lunar/Mars environment is not well known by the crew. Simulators, mockups, 
photos will not give the crew enough information to plan and train for exploration on the lunar 

surfaces. 

VR ADDllcatlon/ Approach: A VE database would be a database of virtual experiences 
experienced from previous mission explorations and/or recordings of explorations done from 
non manned vehicular exploration. These experiences could be used by flight planners to 
determine access to available sites. This type of planning would be done prior to training of the 
Crew Crew members then would be able to retrace the previously explored terrain and plan paths 
for exploration based on the current sites explored. An example might be on the first mission the 
crew searched for Mare sites. The next crew mission is looking for areas that have mature regolith. 
If the crew could experience the previous EVA they may find the site they ® r ® ° 
dpipte areas as possible exploration site. Note: The database is considered to be dynamic 
because it would have to updated after every mission and in between missions from non-manned 

exploration data. 

Benefits: Crew members could experience EVA environment before ' t J? 8 

VE database would help crew train for different seasons, terrain, and tasks. It could redu 
cost and time of training the crew, performing the site selection task, and aid in mapping of lunar 

surfaces. 


Prepared by: 

Benjamin Beberness, 

Lockheed Engineering & Science Co. 

Houston, TX 77058 

Phone: (713) 333-7447 

Fax: (713) 333-6626 

beberness@lock.nasa.gov 

beberness%lock.span@star.stan.edu 
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Task Analysis 


Mission: Exploration (Lunar/Mars) 

Center: Johnson Space Center 

Problem: Analysis of problems that arise real-time is often times accomplished by the ground 
crew. It is sometimes necessary to use existing mockups to simulate the problem to be analyzed. 
These mockups do not always have the fidelity needed to accurately simulate the problem. The 
mockup may need to be reconfigured. 

VR Application/ Approach: Simulate the hardware of the space vehicle in a VR environment. 
The ground crew could reconfigure it to duplicate the problem. The VE could then be used to 
analyze the best approach to accomplish the appropriate task. The VE could then be stored for 
future use. The VR solution could be sent to the VR system on-board for training of the crew 
members. 

Benefits: The VR environment of the space vehicle could be done before the hardware is flown. 
It could be used during the design phase to get the best human factor design. Used during all 
design and construction phases, the VE would be constantly kept updated. It is much less 
expensive and time consuming to redesign a virtual piece of hardware. It would give the crew 
members a better way of understanding the solution to the problem and also give them a chance to 
try the solution on the virtual hardware. 

Benefits: A VR environment would allow faster and less expensive simulation and 
reconfiguration of the appropriate hardware for real-time task analysis. 

Coordination: SP/Man-Systems Division 

Prepared by: 

Marsha Minchew/C95 
Lockheed ESCO 
2400 NASA Road 1 
Houston, TX. 77058 
713-333-6614 
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Crew Health - Medical 


Mission: Exploration (Lunar/Mars) 
Center: Johnson Space Center 


Problem. Crew members on long duration exploration missions will need medical treatment 
during the mission. A medical doctor may not be available onboard or may not be current in the 
treatment procedure for the crew member's illness/injury. 


VR ADDllcatlon/Approach: A treatment procedure such as surgery could be sent 
electronically to the onboard medical attendant and that person could then use a VR system to 
experience the treatment, thereby gaining sufficient knowledge to carry it out onboard. 


Benefits: The onboard medical attendant could be more effective in the treatment of crew 
member illness/injury by receiving such training. 


Savings 

Less People/Time: possibly enables a smaller crew. 

Less Money 
More Effectiveness 

Safety: increases chances of crew member recovery. 

Increased Effectiveness: enhance effectiveness of medical attendant 


Coordination: 

I A/Advanced Initiatives Office 
AP/Office of Public Affairs 
CD/Astronaut Office 
D/EVA and Crew Systems 
DC/Training 

PT4/Software Technology Branch 
SP/Man-Systems 


Prepared by: 

Robert L. Jones, Mail Code C95 
Lockheed Engineering and Sciences 
2400 NASA Road 1 
Houston, Texas 77058 
(713)333-6669 (voice) 
(703)333-6626 (fax) 
bjones@giotto.jsc.nasa.gov 
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Crew Health - Entertainment 


Mission: Exploration (Lunar/Mars) 

Center: Johnson Space Center 

Problem: Entertainment during long duration missions will be an important factor. 

Considerations for entertainment include: space & weight requirements, power requirements, 
cost, challenge for the mind, and flexibility. 

VR Application/ Approach: Virtual reality is making headway in the entertamment field at the 
Dresent time. A VR system could be designed with a great many levels of flexibility and challenge. 
Different VR entertainment systems could be designed to suit different crew members interests. 
The cost, space & weight requirements, and power requirements should be minimal with a VR 

system. 

Benefits: A VR entertainment system would relieve the boredom that free time may ^ring i ^ rin9 
long duration missions. Crew members will be happier if they have a challenging and changing 
form of entertainment. 

Benefits: A VR entertainment system could use a computer that is also utilized for other 
purposes. 

Coordination: SP/Man-Systems Division 


Prepared by: 

Marsha Minchew/C95 
Lockheed ESCO 
2400 NASA Road 1 
Houston, TX. 77058 
713-333-6614 
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Crew Health - Virtual Confidant 


Mission: Exploration (Lunar/Mars) 

Center: Johnson Space Center 

Problem: During long duration missions a problem or circumstance may be encountered that 
makes a crew member want to discuss their feelings with a confidant. 

VR Application/Approach: With a confidant in a VR world, the crew member would be free to 
discuss their feelings about the problem without involving other crew members or the ground 
crew. This virtual confidant would have characteristics of some that the crew member trusts. 

Benefits: Strife or preoccupation with personal matters can cause problems, not only with the 
individual crew member, but also with the team. Some crew members would be happier during 
long duration missions if they had this avenue available to them. Also, it would be less likely that 
they would involve others if there were another avenue to go through. 

Savings: Happier crew membesr are more effective and efficient. 

Coordination: SP/Man-Systems Division 

Prepared by: 

Marsha Minchew/C95 
Lockheed ESCO 
2400 NASA Road 1 
Houston, TX. 77058 
713-333-6614 
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In Situ Training 


Mission: Exploration (Lunar/Mars) 

Center: Johnson Space Center 

Problem: Crew members on extended missions will have to maintain their vehicle systems during 
the mission. They cannot train for all possible maintenance actions prior to the mission. 

VR Appllcatlon/Approach : An onboard VR system could be used for crew training and 
familiarization with maintenance actions they must accomplish. 

Benefits: This could result in reduced preflight training requirements and increased inflight 
maintenance effectiveness. 

Savings: 

Less People/Time: possibly reduces crew training time. 

Less Money 
More Effectiveness 

Safety: reduces safety risk of inflight failures. 

Increased Effectiveness: enhances mission success. 

Coordination: 

lA/Advanced Initiatives Office 
AP/Office of Public Affairs 
CD/Astronaut Office 
D/EVA and Crew Systems 
DC/Training 

PT4/Software Technology Branch 
SP/Man-Systems 

Prepared by: 

Robert L. Jones, Mail Code C95 
Lockheed Engineering and Sciences 
2400 NASA Road 1 
Houston, Texas 77058 
(713)333-6669 (voice) 

(703)333-6626 (fax) 
bjones@giotto.jsc.nasa.gov 
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Planetary Science 

Mission: Exploration (Lunar/Mars) 

Center: Johnson Space Center 

Problem: Crew members on planetary missions will explore the planet surface as much as 
possible. Scientists on Earth will also participate in this exploration. 

VR Application/Approach: A VR system on Earth could allow scientists to more fully 
participate in planetary exploration. 

Benefits: This could result in increased effectiveness in on-site crew members’ exploration of the 
planet surface by utilizing inputs from Earth-based scientists who are also experiencing the 
exploration environment. 

Saving s • 

Less People/Time: possibly reduces crew size needed. 

Less Money 
More Effectiveness 

Increased Effectiveness: enhances mission science return. 

Makes Impossible Possible: enables Earth-based personnel to experience the planet 
surface. 

Coordination: 

I A/Advanced Initiatives Office 
AP/Office of Public Affairs 
CD/Astronaut Office 
D/EVA and Crew Systems 
DC/Training 

PT4/Software Technology Branch 
SP/Man-Systems 

Prepared by: 

Robert L. Jones, Mail Code C95 
Lockheed Engineering and Sciences 
2400 NASA Road 1 
Houston, Texas 77058 
(713)333-6669 (voice) 

(703)333-6626 (fax) 
bjones@giotto.jsc.nasa.gov 
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Shared Experience: Science, Operations, and Education 


Mission: Exploration (Lunar/Mars) 

Center: Johnson Space Center 

Problem: The experience of extended habitation in a sub-g environment will be difficult to 
communicate. Yet that communication will be important to at least four classes of people: mission 
planners responsible for subsequent missions: mission specialists who will train for subsequent 
missions: "earth-bound" scientists and other professionals who need to understand the human 
experience of exploring and working in an extraterrestrial environment; and the general public from 
whom NASA's support ultimately comes. 

VR Appllcation/Approach: A virtual model of the habitat and a portion of the surrounding 
environment can not only provide a faithful model of the spatial layout and the gravitational effects 
but can also be easily changed and can be ported to other computational platforms or shared 
through networking. 

Benefits: Virtual simulation can disseminate information more widely than could be done In any 
other affordable way. Changes can be incorporated relatively easily, as modifications occur 
through time and as new features are discovered through exploration. The portability of software 
will allow planning and familiarization to take place in locations unrestricted by location and 
schedule of a physical simulator. Essentially the same models used for planning and training can 
be made available to scientists for their work and to the general public for their education. 


Coordination: 

lA/Advanced Initiatives Office 
AP/Office of Public Affairs 
CD/Astronaut Office 
D/EVA and Crew Systems 
DC/Training 

PT4/Software Technology Branch 
/ '^P/Man-Systems 



epared by: 

N^rt L. Jones, Mail Code C95 
Xeed Engineering and Sciences 
'l^SA Road 1 
Y Texas 77058 
§669 (voice) 

'§26 (fax) 

*to.jsc.nasa.gov 
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Proficiency Training 

Mission: Exploration (Lunar/Mars) 

Center: Johnson Space Center 

Problem: As space flight becomes increasingly complex and of longer duration, higher levels of 
crew performance in an unfamiliar and stressful environment will be required. Stressors will include 
isolation from familiar work and living environments, potentially high workloads, weightlessness, 
and danger. In addition, due to the anticipated length of these missions, countermeasures for 
maintaining performance at acceptable levels will need to be developed. The current proposal 
seeks to explore the utility of VR technology as a proficiency training tool. 

VR Appllcatlon/Approach: Phase I -- Conduct an analysis of those activities to be performed 
on extended-duration missions and determine those which are likely to degrade due to lack of 
practice. Conduct a media analysis to assess those training objectives best taught through VR 
technology. Implement those objectives through a VR simulation. Phase II -- Conduct attribute 
and performance evaluations to assess the effectiveness of VR technology as a proficiency 
training tool. Phase III -- Demonstrate the use of VR technology as a proficiency training too! by 
developing an abbreviated training module. 

Benefits: Use of this type of embedded training technology can be expected to provide a 
vehicle for maintaining the proficiency of those tasks to be performed on extended missions 
through intermittent virtual training enroute. 

Benefits: Given the space limitations of current spacecraft, VR will provide a less expensive and 
more comprehensive means for ensuring acceptable levels of both individual and crew- 
coordinated task performance. 

Coordination: SP/Man-Systems Division 

Prepared by: 

Manuel F. Diaz, Lockheed ESCO 
2400 NASA Rd 1 
Houston, TX 77058 
713-333-7129 
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After-the-Fact Analysis, Accident or Event Reconstruction 

Mission: Space Transportation System (STS) 

Center: Johnson Space Center 

Problem: After problems arise aboard the Space Shuttle, methods used to assess the situation 
are often expensive, time-consuming, and inadequate for grasping the intricacies and complexity 
of the situation. 

VR Application/Approach: Use the extensive available data to construct a VR and allow 
investigators and problem-solvers to more than review the data, but to actually immerse 
themselves in the scene and evaluate the situation as an observer or even participant. 

Benefits: The added insight available to the investigators would allow a faster interpretation of 
the situation and result in faster corrective recommendations. This approach, rather than setting 
up full-size mockups for evaluation, could result in much less time and expense necessary for real- 
time problem-solving during a mission and could lead to better, more successful methods of 
dealing with the problem. 

Coordination: SP/Man-Systems Division 

Prepared by: 

Carlos Sampaio/C95 
Lockheed ESCO 
Houston, TX 
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Hubble Space Telescope Maintenance/Repair 

Mission: Space Transportation System (STS) 

Center: Johnson Space Center 

Problem: Ground-based training fro Hubble Space Telescope maintenance and repair is 
currently insufficient. No high fidelity mock up of the telescope exists on the earth, making it 
difficult to train for maintenance and repair work. 

Description: NASA Hubble photo databases are currently available and can be used to create a 
visualization of the Hubble Space Telescope. Additionally, a Shuttle Extra Vehicular Activity can 
be simulated to effect Hubble repair actions, this virtual trainer would allow the astronaut to practice 
the sequence of actions required to successfully perform a Hubble repair mission. 

Benefits: Provide currently unavailable training; reduce the time and cost required to train for 
HST maintenance and repair; enhance safety. 

Coordination: 

CB/Astronaut Office 
DG/Training Division 
DT/Space Station Training Division 
ER/ Automation & Robotics Division 
PT4/Software Technology Branch 

Prepared by: 

Beth Holewinski/DT3 
NASA/Johnson Space Center 
Houston, TX 77058 
713-283-8131 (voice) 

713-283-8126 (FAX) 
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EVA/RMS Training and Procedures Development for HST 

Repair 

Mission: Space Transportation System (STS) 

Center: Johnson Space Center 

Problem: Scenarios, such as the Hubble Space Telescope (HST) Repair Mission, require a great 
deal of coordinated interaction between the EVA crewmen and the Remote Manipulator System 
(RMS) operator. Development of detailed timelines and procedures to adequately train for such a 
mission is compromised by the fact that for ground based training there is no facility that can fully 
integrate EVA and RMS operations. Neutral Buoyancy facilities are available to simulate EVA 
activities, but no facility is large enough to employ a fully functional RMS; while computer graphics 
generated or hydraulically operated simulations of the RMS are used to train RMS operators, but 
provide no direct interaction with the EVA crew. 

Description: A Virtual Reality system provides a relatively inexpensive and readily adaptable 
method for choreographing and integrating EVA tasks with RMS tasks to arrive at more realistic 
timelines and procedures. Each EVA crewman would be provided with a VR “helmet and gloves’ 
to allow him to visually assess and interact with the movements of the other EVA crewman, the 
RMS, the HST and the orbiter payload bay configuration. The system could also be used to 
evaluate acceptable rates at which to maneuver the RMS while an EVA crewman is positioned on 
the Manipulator Foot Restraint (MFR). This system will not replace any of the current simulation 
facilities, but will effectively integrate multiple part task trainers. The same scenario is applicable to 
any mission involving coordinated EVA and manipulator operations whether they be orbiter based 
or station based. 

Coordination: 

CB/Astronaut Office 
DF/Mechanical and Crew Systems 
DG/Training 

ER/Automation and Robotics 

Prepared by: 

David Homan/ER 
NASA / Johnson Space Center 
Houston, Texas 77058 
713-483-8089 (voice) 

713-483- 3204 (fax) 
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Crew Training for Satellite Retrieval and/or Repair 


Mission: Space Transportation System (STS) 
Center: Johnson Space Center 


Problem: Ground-based systems for astronaut training cannot provide complete fidelity in the 
area of dynamic response to impulses imparted by crew and/or the Remote Manipulator System 
(RMS). Ground-based training systems are also expensive to build, maintain, and operate. Such 
systems may require large number of support personnel and may have limited access due to 
scheduling constraints. 


Description: Simulation-based models of satellites are commonly-available for both engineering 
and operations development. These models may be modified to include sufficient dynamic 
behavior to support crew training for proximity operations, grapple, retrieval, repair, and 
redeployment A VE utilizing such models would provide relatively low-cost, unlimited training 
experiences to astronauts as EVA procedures are developed and pre-flight training is conducted. 
Coupling the VE to an Intelligent-Computer Aided Training (ICAT) system would further reduce 
manpower and cost requirements, allowing astronauts to train independent of training personnel 
and facility availability. 


Benefits: This approach would provide training unavailable in existing simulators, reduce training 
costs, and reduce training time. 


Coordination: 

CB/Astronaut Office 
DF42/EVA & Crew Systems 
DG/Training Division 
DT/Space Station Training Division 
ER/Automation & Robotics Division 
PT4/Software Technology Branch 
SP/Man-Systems Division 

Prepared by: 

R. Bowen Loftin/Mail Code PT4 
NASA/Johnson Space Center 
Houston, TX 77058 
713-483-8070 (voice) 
713-244-5698 

bloftin@gothamcity.Jsc.nasa.gov 
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EVA Operations Development 

Mission: Space Transportation System (STS) 

Center: Johnson Space Center 

Problem: EVA operations development are currently performed with "pencil and paper" or with 
the use of expensive and scarce simulator resources. 

Description: Virtual environment technology would provide those developing EVA operational 
procedures with a relatively inexpensive and accessible tools for primary operations development, 
verification, and the exploration of options. The speed with which applications can be developed 
and modified would also permit this approach to support real-time operations development during 
missions. 

Benefits: Reduce the time and cost required to develop EVA operations; enhance safety; 
support real-time operations development in response to mission problems/challenges. 

Coordination: 

CB/Astronaut Office 
DF42/EVA & Crew Systems 
DG/Training Division 
DT/Space Station Training Division 
ER/Automation & Robotics Division 
PT4/Software Technology Branch 
SP/Man-Systems Division 

Prepared by: 

R. Bowen Loftin/PT4 
NASA/Johnson Space Center 
Houston, TX 77058 
713-483-8070 (voice) 

713-244-5698 

bloftin@gothamcity.jsc.nasa.gov 
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RMS Training 

Mission: Space Transportation System (STS) 

Center: Johnson Space Center 

Problem: The Shuttle Mission Simulator (SMS) at JSC is used for crew on-orbit training. The 
SMS currently does not provide complete fidelity in the area of dynamic response to impu se 
imparted by the Remote Manipulator System (RMS). 

Description: The SMS provides a high fidelity crew station and state-of-the-art out-the-window 
and CCTV visual simulation using Evans & Sutherland ESIG-3000 image generators. 

Enhancement of simulator math models to include accurate dynamic behavior of free-flying 
Davloads interacting with the RMS would maximize the effectiveness of the SMS f ° r . cr ®w training. 
These math models could be adapted directly from application software developed for VE part-task 

trainers at little additional cost. 


Coordination: 

CB/Astronaut Office 
DG/Training Division 

DK/Space Shuttle Ground Systems Division 
DT/Space Station Training Division 
ER/ Automation & Robotics Division 
PT4/Software Technology Branch 


Prepared by: 

Arthur M. Gorski 
The MITRE Corporation 
1120 NASA Road One 
Houston, TX 77058 
713-333-0980 voice 
713-333-2813 fax 
JSC Mail Code PS/MITRE 
agorski@mitre.org 
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Near-Term VR Applications in Spacelab 

Mission : Spacelab on Space Transportation System (STS) 

Center: Marshall Space Flight Center 

Spacelab is a modular laboratory facility that is carried to and from orbit by the Space Shuttle. It includes a 
pressurized module (short and long configurations) and pallets that can be used in various combinations. 

It provides a shirt-sleeve, well-equipped laboratory environment where scientists can conduct 
investigations in a variety of disciplines, including material sciences, life sciences, astronomy, physics, etc. 

Objective: The planning and preparation for a Spacelab flight begins years before the actual mission 
flies. Payload experiments must be defined and the equipment designed and fabricated. Operational 
concepts must be developed and procedures defined. The Individual payloads must be analytically, as 
well as physically, integrated into the Spacelab system. Individual and integrated payload training, for both 
the crew and the Payload Operations Control Center (POCC) cadre, must take place. During the flight, 
significant ground support resources are expended on payload operations support, science monitoring, 
replanning, and Fault Detection, Isolation, and Recovery (FDIR). 

Each of the planning, preparation, and execution activities present opportunities for application of VR 
technologies and techniques. Examples of applications that can utilize existing VR technology and 
capabilities can be found in the chapter on current VR activities. In large part, the current activities are 
focused on validating VR as a design and operations analysis tool. This section will propose specific 
applications of VR technologies and techniques that initially focus on the continued validation of classes 
of applications, but eventually evolve into strict application of VR as a design and operations analysis and 
support tool. The first two applications can be accomplished with the existing VR technology capabilities, 
though each is limited, for the most part, to visualization with minimal engagement of the object behavior 
and dynamics attributes. The other proposed applications require enhancements over existing VR 
technology and capabilities, particularly in the area of object behavior and dynamics attributes. As these 
enhancements come on line, future Spacelab flights (i.e., those occurring within the next 3-4 years) will 
benefit. 

VR Appllcatlon/Approach: The first proposed application is related to initial crew and POCC cadre 
training. As a new crew or cadre member is assigned to a Spacelab mission there is a familiarization phase 
for both mission independent Spacelab systems and capabilities and mission dependent payload 
systems and capabilities. 

Early in the mission planning and development process, this training is accomplished in the classroom and 
through Spacelab systems and mission documentation. The full-scale Payload Crew Training Complex 
(PCTC) training mock-up and simulators are not yet available. For personnel assigned later, the PCTC 
training mock-up may be in place, but access may be limited due to simulator development and training 
activities. 

In either case there is a period during which the newcomer must quickly assimilate a large amount of 
information into his or her concurrently evolving schema or mental model of Spacelab. A tour through a 
virtual Spacelab may initialize the newcomer and provide insights into system functionality and capabilities. 
If successful, this could provide a basis for a more accelerated training program and a better integrated 
understanding of Spacelab systems and payloads. 

The essential feature of this application is one or more Virtual Spacelab Modules (VSLMs). Depending on 
the focus of the "lesson", there may be several VSLMs, each configured to support that lesson objective. 
For example, Spacelab Program Overview may use standard Spacelab systems in both the long and short 
modules and perhaps even the pallets only configuration. Mission specific training could use a Spacelab 
systems only VSLM and/or an integrated systems/payload VSLM. In addition, each system and payload 
could be "exploded" to permit visualization of its constituent components and their interrelationships. 

The exact details for a particular VSLM would depend on specific training objectives and existing VR 
technology capabilities and limitations. Validation of this application will be based primarily upon subjective 
data gathered through questionnaires and structured. As the VR technology capabilities are enhanced, 
this application can be expanded to include additional, more complex training objectives. 
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A related "real world" application, utilizing existing VR technology capabilities, involves using VSLMs 
during the last nine-to-six months before launch. There are always late changes to on-board stowage. As 
changes are made, the PCTC Training mock-up is updated. It is desirable to allow the crew the 
opportunity to tour the mock-up to "see" the latest stowage configuration. This helps to "internalize" the 
location of items within the Spacelab module. Unfortunately, as the launch date approaches, access to 
the crew becomes more and more limited, particularly during the last three months. 


A VSLM with the updated stowage configuration would enable a more convenient, even remote, method 
to "visualize" changes in stowage locations. Updated VSLM files could even be electronically transmitted 
to the Johnson Space Center (JSC) for the crew to "tour" on the JSC VR system. Validation of this 
application, like the previous real-world application, will be based primarily upon subjective data gathered 
through questionnaires and structured. 

This ability to electronically transfer Virtual Worlds (VWs) further enhances the familiarization/initialization 
training application discussed above. In fact, another existing VR technology capability can enhance both 
of the "real world" Spacelab applications. Using both the MSFC and JSC VR systems simultaneously, the 
users could enter and interact within the same VSLM at the same time, even though they are physically 
located in different states! This would permit, for example, a "tour guide" for the Spacelab Program 
Overview or a Mission Specialist accompanied by the stowage manager or a Payload Specialist for the 
stowage "walk-thru". 

Two major enhancements are required in object behavior and dynamics attributes for more advanced 
applications of VR technology. These are incorporation of an anthropometric model and a physics 
properties simulator reflecting physical laws concerning motion and collisions. 

The anthropometric model should include link lengths to reflect a broad anthropometric design range 
(e g 5th percentile female through the 95th percentile male) and realistic joint range-of-motion 
capabilities and constraints. The physics properties simulator should include realistic linear and angular 
acceleration/velocity and kinetic energy transfer. 


As these enhancements come on line, the training application discussed above, for example, can be 
expanded to cover more demanding training objectives. However, instead of discussing an expanded 
training application, another application will be proposed, although the discussion applies to the training 
application as well. 

A demanding and comprehensive application for VR is support of unplanned Inflight Maintenance (IFM). 
That is, subsets of the features and VR capabilities required to support this application are used in a vanety 
of other applications. Support to unplanned IFM requires Human Factors analyses (e.g., viewing, reach, 
and dynamic work envelope analyses), operations development, training, mission support, and even 
simultaneous participation by physically separated users in the same VE. 


An example of an unplanned IFM occurred on Spacelab 3. This actual Spacelab mission experience wilf 
also be used for comparison in the validation of this application. The goal would be to actually recreate the 
IFM environment and operation, then compare this virtual IFM experience with the actual flight experience. 
This would include reference to video and audio recordings of the on-board operation, written logs, and 
participation of the actual Spacelab crew involved in the IFM operation. 


During Spacelab 3, the Drop Dynamics Module (DDM) developed a problem with a power supply module. 
No procedures or plans had been developed pre-mission for this particularly contingency. No spare 
power supplies were stowed. It was decided to remove an in-service power supply, from another on- 
board system, and use it in the DDM. Procedures had to be developed and validated on the ground and 
approved by both MSFC and JSC before uplink to the crew. The procedure required removal of the rack 
front panel before the Payload Specialist (PS) entered head first. Only his legs remained visible outside of 
the rack. Inside the cramped rack interior, the PS, successfully exchanged the power supply modules and 
continuation of the science objectives resumed. 


It is anticipated that enhanced VR will be capable of supporting many of the activities and analyses that 
occurred on the ground in support of this unplanned IFM, Viewing analyses, reach envelope analyses, 
and, with an incorporated anthropometric model, dynamic work envelope analyses can be achieved 
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concurrently with procedure development. Although much of this can be done in an engineering mock- 
up, VR offers several unique capabilities. 

First VR could provide a timely and safe method to enable the various advantages and disadvantages of 
reaching and maneuvering in a microgravity environment. This includes body attitudes and positions 
difficult to recreate in a one-G environment. This would be superior to existing methods for simulating 
micro-gravity because existing methods can not be used in a timely manner and are of limited durations 
(KC-135) or require ancillary equipment (Neutral Buoyancy Simulator) that can interfere with operations in 

restricted volumes. 

Second, VR would permit anthropometric sizing to reflect the dimensions of the on-board crew. This is 
particularly useful for operations being planned in relatively tight spaces. 

Once the DDM IFM procedures had been developed and validated, MSFC and JSC had to approve the 
operation before it could be implemented. VR would offer the mission and payload managers the ability to 
visualize the procedure and environment to gain a faster and more in-depth understanding of the 
operation This could be accomplished while the managers are sitting at their consoles in the controf 
center. Further, managers, at both centers, could enter the VW, simultaneously, to review and discuss 
the operation. This capability for direct mission support would be unprecedented, though the possibilities 
are not limited unplanned IFM. 

Pre-mission operations development and validation could also be carried out in the same manner, even 
thouqh the rapid turn-around capability of VR is not necessarily a requirement. Pre-mission crew training 
could use the same VWs developed to support procedure development. This would prove particularly 
beneficial for operations where the various advantages and disadvantages of reaching and maneuvering 
in a microgravity environment make a difference. 

The second United States Microgravity Laboratory (USML-2), scheduled for launch on May 6, 1995, is 
proposed as the first "full-up" VR applications Spacelab. VR will have been applied, for selected analyses, 
on earlier Spacelab missions, but USML-2 will be the first mission to which all of the techniques and 
validated tools, resulting from the previous phases, will be applied. 

USML-2 is the second in a series of Spacelab flights that focus on microgravity materials processing 
technology, science, and research. These USML missions emphasize technology development within 
the United States to develop Space Station applications. 

USML-2 was selected primarily because it is the first full module mission to which the validated VR toofs will 
be "on-line" during the appropriate phase of the program. That is, for example, validated Human Factors 
analytical tools during the design phase, Spacelab module familiarization trainer early in the training phase, 
stowage trainer in the last months before launch, and an enhanced set of tools for real-time mission 
support. 

A secondary reason USML-2 was chosen is because USML-1 flies June 3, 1992. It is anticipated that a 
portion of USML-2 will be of USML-1 experiments. Thus, USML-1 provides actual flight experiences upon 
which to base USML-specific validation studies. Further, it might provide insights into potential mission 
peculiar IFMs. This could help refine the USML-2 VW requirements. 

Benefits: The potential benefits from the application of VR technologies and techniques to Spacelab 
planning, preparation, and execution are significant. More efficient utilization of constrained resources 
can be realized. 

Viewing analyses, reach envelope analyses, and, with an incorporated 1 

work envelope analyses can be achieved concurrently with procedure development. VR can provide a 
timely and safe method to enable the various advantages and disadvantages of reaching and 
maneuvering in a microgravity environment. This would be supenorto existing methods for simulating 
mfcro^ravSy because existing methods can not be used in a timely manner and are of l.mrted dun aborts. 
Even where the KC-135 and/or the NBS are appropriate, pnor utilization of virtual mockups cai 
more efficient use of these micro-gravity simulators. Hardware and operations design can be more mature, 
resulting in fewer and/or more productive simulator sessions. 
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Pre-mission operations development and validation can utilize VR. Pre-mission crew training could use 
the same VWs developed to support procedure development. This would prove particularly beneficial for 
operations where the various advantages and disadvantages of reaching and maneuvering in a 
microgravity environment make a difference. 

A VSLM with the updated stowage configuration can enable a more convenient, even remote, method to 
"visualize" changes in stowage locations. Using both the MSFC and JSO VR systems simultaneously, the 
users could enter and interact within the same VSLM at the same time. This can permit, for example, a 
"tour guide" for the Spacelab Program Overview or a Mission Specialist accompanied by the stowage 
manager or a Payload Specialist for the stowage "walk-thru". 

Once unplanned IFM procedures have been developed and validated, MSFC and JSC must approve the 
operation before it can be implemented. VR would offer the mission and payload managers the ability to 
visualize the procedure and environment, while sitting at their consoles in their respective control centers. 
Further, managers, at both centers, could enter the VW, simultaneously, to review and discuss the 
operation. This capability for direct mission support would be unprecedented, though the possibilities are 
not limited unplanned IFMs. 

Prepared by: 

Joseph P. Hale 

Man/Systems Integration Branch 
NASA Marshall Space Flight Center 
Marshall Space Flight Center, AL 3581 2 
(205) 544-2193 
jphale@nasamail.nasa.gov 
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Crew Health and Performance 

Mission: Space Transport System (STS) and Space Station 
Center: Johnson Space Center 

Problem: Resolution of space motion sickness, and improvements in spatial orientation, posture 
and motion control, and compensatory eye movements occur as a function of neurosensory and 
sensory-motor adaptation to microgravity. These adaptive responses, however , are inappropriate 
for return to Earth and can result in postural, gait, and visual instabilities as well as disturbances in 
eye-hand coordination. As mission duration increases, these neurosensory and sensory-motor 
disturbances are expected to be magnified and may impact crew safety during the entry/landing 
and egress phases of the mission. 

VR Application/Approach: A head-mounted system, configured with head position or rate 
sensors, could be used to present and desired visual environment; a dataglove and or joystick 
could be used for performance of predetermined tasks. The head and limbs of the user could be 
loaded to simulate a 1 g gravitational force. A set of tasks requiring eye-head and eye-hand 
coordination, and possibly locomotion could be designed with visual and tactile or force feedback 
correct for a 1g environment. Crew members would practice various task scenarios prior to their 
return to Earth. Such a system may include instrumentation for recording eye and limb 
movements. 

Benefits: A VR system and training scenarios are being developed for preadapting astronauts to 
microgravity and to maintain adaptation to Earth (i.e., to produce dual-adapted states). This system 
is called the device for orientation and motion environments (DOME) - preflight adaptation trainer 
(PAT). For long-duration STS and Space Station missions, an inflight VR system could help crew 
members maintain appropriate neurosensory and sensory-motor responses for the 1g Earth 
environment. 

Coordination: SD/Medical Sciences Division 

Prepared by: 

Dr. D. Harm/SD51 1 
NASA JSC 
713 483 7222 
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SAFER Engineering Test and Development 

Mission: Space Transportation System (STS) / Space Station 
Center: Johnson Space Center 

Problem: The Simplified Aid For EVA Rescue (SAFER) project is developing a “ mini-backpack" 
for use by EVA crew members at Space Station as a means of self-rescue in the event that they 
inadvertently become untethered from the structure. Control algorithm development and testing 
for the SAFER are performed on ground based simulators and have to take into account all the 
various control modes and operational scenarios to validate the functional design. Stereo visual 
cues, with a wide field-of-view, are a major contributor to the operation and thus the design of the 
control system. Motion base simulations for this type of development are cost prohibitive. 

Description: A VR system provides the display technology required for crew evaluation of the 
SAFER system’s ability to perform its desired function, as well as a means of investigating 
additional operating scenarios or control system strategies. The VR system also provides a realistic 
training environment for future users of the SAFER. 

Coordination: 

CB/Astronaut Office 
ER/Automation and Robotics 

Prepared by: 

David Homan/ER 
NASA / Johnson Space Center 
Houston, Texas 77058 
713-483-8089 (voice) 

713-483- 3204 (fax) 
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Manipulator Systems Training 

Mission: Space Station 
Center: Johnson Space Center 

Problem: The Space Station Training Facility (SSTF) at JSC will include a cupola trainer for crew 
out-the-window training tasks. The SSTF is not planned to provide complete fidelity in the area of 
dynamic response to impulses imparted by the manipulator systems controlled from the Space 
Station cupola. 

Description: The SSTF will provide a high fidelity crew station and state-of-the-art out-the- 
window and CCTV visual simulation using Evans & Sutherland ESIG-3000 image generators. 
Enhancement of simulator math models to include accurate dynamic behavior of free-flyers 
interacting with the station manipulators would maximize the effectiveness of the SSTF for crew 
training. These math models could be adapted directly from application software developed for VE 
part-task trainers at little additional cost. 

Coordination: 

CB/Astronaut Office 

DJ/Space Station Ground Systems Division 
DT/Space Station Training Division 
EFt/Automation & Robotics Division 
PT4/Software Technology Branch 

Prepared by: 

Arthur M. Gorski 
The MITRE Corporation 
1120 NASA Road One 
Houston, TX 77058 
713-333-0980 voice 
713-333-2813 fax 
JSC Mail Code PS/MITRE 
agorski@mitre.org 
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Space Station Construction 


Mission: Space Station 
Center: Johnson Space Center 

Problem: The Space Systems Automated Integration and Assembly Facility (SSAIAF) at JSC will 
be used for on-orbit assembly procedures development and training. The SSAIAF is intended to 
provide good fidelity in the area of dynamic response between free-flying structures, the shuttle 
orbiter, and manipulator systems using a high-fidelity real-time simulation driving mechanical 
systems. Currently, there are no plans for a system to validate results from SSAIAF simulations. 

Description: A VE component dedicated to the SSAIAF facility could reduce risk by providing a 
check against hardware system results and a visual view of scenes not possible to represent 
accurately with the test- oriented hardware. 

Coordination: 

CB/Astronaut Office 
DT/Space Station Training Division 
ER/Automation & Robotics Division 
PT4/Software Technology Branch 

Prepared by: 

Arthur M. Gorski 
The MITRE Corporation 
1120 NASA Road One 
Houston, TX 77058 
713-333-0980 voice 
713-333-2813 fax 
JSC Mail Code PS/MITRE 
agorski@mitre.org 
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In Situ Training 


Mission: Space Station 
Center: Johnson Space Center 

Problem: Long-duration missions, such as the Space Station, will require refresher crew training 
for infrequently performed tasks. Simulators containing hardware elements cannot be flown to 
support such training. 

Description: Virtual environment technology would provide access to a complete simulation for 
crew training while onboard the Space Station. Such simulation could encompass both EVA and 
IVA tasks and would be especially effective for infrequently performed tasks. 

Benefits: Provide training capabilities not available through other mechanisms; enhance training 
and probability of mission success. 

Coordination: 

CB/Astronaut Office 
DG/Training Division 
DT/Space Station Training Division 
ER/Automation & Robotics Division 
PT4/Software Technology Branch 
S P/Man-Systems Division 

Prepared by: 

R. Bowen Loftin/Mail Code PT4 
NASA/Johnson Space Center 
Houston, TX 77058 
713-483-8070 (voice) 

713-244-5698 

bloftin@gothamcity.jsc.nasa.gov 
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Crew Medical Restraint System 


Mission: Space Station 
Center: Johnson Space Center 

Problem: Simulating transport of an injured crew member on the Crew Medical Restraint Systern 
(CMRS) from Space Station to the orbiter mid-deck through the transfer tunnel in 1 g. The difficulty 
arises in performing the simulation when passing the restrained injured crew member through the 
90 degree turn in the tunnel prior to entry into the orbiter airlock. 


VR Applicatlon/Approach: VR could be used to verify whether transport of the restrained 
crew member with attending Crew Medical Officer(s) (CMO) and medical equipment is actually 
possible. Furthermore, VR could be used to determine the optimal configuration for passing the 
patient and CMO(s) through the tunnel and into the orbiter. 

Benefits: This application of VR would simplify a logistically difficult Health Maintenance Facility 
(HMF) simulation scenario. Other possible options for performing this simulation have various 
limitations associated with them. For example if CHeCS were to perform this simulation in the 
WETF they would have to deal with the divers oxygen tanks which would probably add a 'eve lot 
difficulty to passing through the tunnel and thus would not provide an accurate simulation of the 
transport scenario or the volume envelope of the crew members. When CHeCS performs this 
simulation on the KC-135 the period of weightlessness is not long enough to definitive^ 
determine if there should be two patient attendees as opposed to one. In addition the KC-135 is 
not larqe enough to create an actual mockup of the tunnel with the 99 degree turn which is one of 
the main problems in the simulation. The only other possible alternative to assess a complete 
solution is to simulate this patient transport scenario on the shuttle which would be costly and 
probably could not be manifested during the development period of the CMRS.^ Consequently 
the use of Virtual Reality in this simulation would probably have safety and reliability benefits as well 
as an increased effectiveness in the design of the CMRS and the actual development of the 
operational scenario. 


Coordination: 

lA/Advanced Initiatives Office 
AP/Office of Public Affairs 
CD/Astronaut Office 
D/EVA and Crew Systems 
DC/Training 

PT4/Software Technology Branch 
SP/Man-Systems 


Prepared by: 

Robert L. Jones, Mail Code C95 
Lockheed Engineering and Sciences 
2400 NASA Road 1 
Houston, Texas 77058 
(713)333-6669 (voice) 
(703)333-6626 (fax) 
bjones@giotto.jsc.nasa.gov 
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Space Station Operations (IVA and EVA) 

Mission: Space Station 
Center: Johnson Space Center 

Problem: EVA and IVA operations development are currently performed with "pencil and paper” 
or with the use of expensive and scarce simulator and/or mockup resources. 

Description: Virtual environment technology would provide those developing Space Station 
operational procedures with a relatively inexpensive and accessible tools for primary operations 
development, verification, and the exploration of options. The speed with which applications can 
be developed and modified would also permit this approach to support real-time operations 
development during missions. 

Benefits: Reduce the time and cost required to develop Space Station EVA and IVA operations; 
enhance safety; support real-time operations development in response to mission * 
problems/challenges. 

Coordination: 

CB/Astronaut Office 
DG/Training Division 
DT/Space Station Training Division 
ER/Automation & Robotics Division 
PT4/Software Technology Branch 
SP/Man-Systems Division 

Prepared by: 

R. Bowen Loftin 
Mail Code PT4 

NASA/Johnson Space Center 
Houston, TX 77058 
713-483-8070 (voice) 

713-244-5698 

bloftin@gothamcity.jsc.nasa.gov 


NASA VE R&T 


112 


Near-Term VR Applications in Space Station 


Mission : Space Station 
Center: Marshall Space Flight Center 

Space Station will be a permanently manned orbiting facility that will seiye as a permanent observatory and 
provide research capabilities in such disciplines as fluid physics, materials sciences, combustion, 
biotechnology, life sciences and technology. The Space Station Program is currently entering the critical 
design phase where major decisions regarding design and operations are being made. As the operations 
and utilization phase approaches, planning for payload operations is beginning. This includes payload 
experiment development, payload analytical integration, training and operations support. 

Objective: During the critical design phase, intra- and inter-system design is progressing to an ever finer 
level of detail. Throughout this phase, analytical studies compare alternate candidate design solutions 
and evaluate the consequences of design decisions. Operations concepts, for both system operation 
and maintenance, are also maturing during this phase. Operations development is accomplished hand-in- 
hand with hardware development, both in turn being refined during each iteration to approach an 
"optimal" man-machine system. This hardware and operations development can benefit through the 
application of VR technologies and techniques. 

The planning and preparation for an Space Station increment begins years before the actual mission flies. 
Payload experiments must be defined and the equipment designed and fabricated. Operational 
concepts must be developed and procedures defined. The individual payloads must be analytically, as 
well as physically, integrated into the Space Station system. Individual and integrated payload training, for 
both the crew and the Payload Operations Integration Center (POIC) cadre, must take place. During the 
increment, significant ground support resources are expended on payload operations support, science 
monitoring, replanning, and Fault Detection, Isolation, and Recovery (FDIR). Each of the planning, 
preparation, and execution activities present opportunities for application of VR technologies and 
techniques. 

Examples of applications that can utilize existing VR technology and capabilities can be found in the 
chapter on current VR activities. In large part, the current activities are focused on validating VR as a _ 
design and operations analysis tool. This section will describe potential applications of VR technologies 
and techniques as a design and operations analysis and support tool. Many applications can be 
accomplished with the existing VR technology capabilities, though each is limited, for the most part, to 
visualization with minimal engagement of the object behavior and dynamics attributes. Other applications 
require enhancements over existing VR technology and capabilities, particularly in the area o* object 
behavior and dynamics attributes. As these enhancements come on line, future Space Station 
development, and operations will benefit. 

VR Application/Approach: Human Factors issues and considerations in hardware and operations 
development present a large class of potential VR applications. VR technologies and techniques 
currently provide some limited macro- and micro-ergonomic analytical tools for consideration of 
operational, viewing and reach envelope requirements, in both one-gravity and microgravity 
environments. 


An algorithm has been developed to rescale user anthropometric attributes to any desired virtual 
anthropometry. Thus, a 95th percentile male could view and reach as a virtual 5th percentile female and 
vice-versa. Further, a technique has been developed where the user inside a virtual module can 
manipulate the attitude of that module, as a whole, while "grabbing" a handrail, giving the egocentric 
perception of microgravity mobility. This can provide some of the various advantages and disadvantages 
of reaching and maneuvering in microgravity. 

Combined with scaleable user anthropometry attributes, macro-ergonomics analyses for the topological 
design of work areas can consider what one is able to see from a variety of eye reference points using a 
range of virtual anthropometric sizes. These analyses can include operationally-driven components such 
as translation paths among the various wori<siles. Micro-ergonomics analyses for the spatial layout of 
workstations can consider what one is able to see from a variety of eye reference points and on what one 
is able to touch from a variety of shoulder and seat reference points and/or foot restraint locations. 
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VR technologies and techniques can be applied in the Space Station Program, starting during the critical 
design phase. Many analyses that use Fomecor mockups, the KC-135, or the Neutral Buoyancy 
Simulator are candidates for VR. It is not that VR would completely replace these other technologies and 
techniques, but that it adds another tool to the analytical toolkit. 

In some instances, VR might be considered for use in an analysis that would have otherwise not be 
undertaken. Resources (time, people, materials, etc.) required for a "standard" simulation or mock-up 
analysis may be greater than the expected return. In this case VR, due to its relatively low utilization costs, 
would surpass the cost/benefit ratio threshold and enable an analysis that would have otherwise been 
forgone. 

Similarly, VR can enhance and enable more effective utilization of standard simulations and mock-up 
analyses. By preceding these analyses with preliminary VR analyses, both the hardware and operations 
can be refined so that the return from the standard analyses is increased. This is accomplished by either 
reducing the magnitude or number of standard analyses and/or improving the fidelity of those analyses 
with a more mature design. 

Because the Virtual Worlds (VWs) are nothing more than computer files, design changes can be done 
more quickly and more candidate configurations can be subsequently analyzed than is currently possible 
with existing, "standard" Human Factor tools (e.g., Fomecor mockups). 

The list of potential VR critical design phase assessments/analyses is extensive. Only a few examples are 
given here, these include maintenance access (e.g., within a rack, behind a standoff, etc.), restraint and 
mobility aid location, rack pivot/removal/, module topology and color selections, and logistics module 
access and resupply operations (e.g., translation routes, mobility aids, etc.). Design of ground support 
and processing facilities and operations can also benefit through VR utilization. Using VR to visualize and 
interact with various Payload Operations Integration Center (POIC) configuration options, for example, 
provides a design analytical capability that is not otherwise possible. 

Many of the design analytical applications for the SSFP critical design phase can be applied equally 
effectively to the design and integration of payloads. A more general operations and utilization application 
is related to initial crew and POIC cadre training. As a new crew or cadre member is assigned to a Space 
Station increment there is a familiarization phase for both increment independent Space Station systems 
and capabilities and increment dependent payload systems and capabilities. 

Early in the increment planning and development process, this training will be accomplished in the 
classroom and through Space Station systems and increment documentation. The full-scale Payload 
Training Complex (PTC) training mock-up and simulators will not yet be available. For personnel assigned 
later, the PTC training mock-up may be in place, but access may be limited due to simulator development 
and training activities. 

In either case there is a period during which the newcomer must quickly assimilate a large amount of 
information into his or her concurrently evolving schema or mental model of Space Station. A tour through 
a virtual Space Station may initialize the newcomer and provide insights into system functionality and 
capabilities. If successful, this could provide a basis for a more accelerated training program and a better 
integrated understanding of Space Station systems and payloads. 

A related application, utilizing existing VR technology capabilities, involves using a virtual Space Station 
during the last nine-to-six months before an increment. As with Spacelab, it is anticipated there will be late 
changes to on-board stowage. As changes are made, the PTC Training mock-up will be updated. It is 
desirable to allow the crew the opportunity to tour the mock-up to "see" the latest stowage configuration. 
This helps to "internalize" the location of items within the Space Station module. Unfortunately, as the 
launch date approaches, access to the crew becomes more and more limited, particularly during the last 
three months. 

A virtual Space Station with the updated stowage configuration would enable a more convenient, even 
remote, method to "visualize” changes in stowage locations. Updated VW files could even be 
electronically transmitted to the Johnson Space Center (JSC) for the crew to "tour on the JSC VR 
system. 
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maneuvering in a microgravity environment. This would be superior to existing methods for simulating 
micro-gravity because existing methods can not be used in a timely manner and are of limited durations. 
Even where the KC-135 and/or the NBS are appropriate, prior utilization of virtual mockups can result in 
more efficient use of these micro-gravity simulators. Hardware and operations design can be more mature, 
resulting in fewer and/or more productive simulator sessions. 

Pre-increment operations development and validation can utilize VR. Pre-increment crew and cadre 
training could use the same VWs developed to support procedure development. This would prove 
particularly beneficial for operations where the various advantages and disadvantages of reaching and 
maneuvering in a microgravity environment make a difference. 

A virtual Space Station with the updated stowage configuration can enable a more convenient, even 
remote, method to "visualize" changes in stowage locations. Using both the MSFC and JSC VR systems 
simultaneously, the users could enter and interact within the same VW at the same time. This can permit, 
for example, a "tour guide" for the Space Station Program Overview or a Mission Specialist accompanied 
by the stowage manager or a Payload Specialist for the stowage "walk-thru". 

VR can support operations development and validation during an increment. VR would offer the mission 
and payload managers the ability to visualize the procedure and environment. Both could enter the VW, 
simultaneously, to review, discuss, and approve the operation. This capability for direct mission support 
would be unprecedented. 

Prepared by: 

Joseph P. Hale 

Man/Systems Integration Branch 
NASA Marshall Space Flight Center 
Marshall Space Flight Center, AL 3581 2 
(205) 544-2193 
jphale@nasamail.nasa.gov 
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Near-Term VR Applications in the Design of the 
Advanced X-ray Astrophysics Facility 

Mission : The Great Observatory series 
Center: Marshall Space Flight Center 

The Advanced X-ray Astrophysics Facility (AXAF) will serve as a permanent X-ray observatory for studying 
IJtfXSmena m S? evolution and the structure of active galaxies. Tl* AXAF Program s : currently 
beina redefined. Two spacecrafts will be developed and launched separately. AXAF-I (Imaging) willbe 
launched aboard the Space Shuttle and placed in a "parking orbit" to be later boosted, with a "kick-motor, 
to probably a 1 0K by 100K km orbit. The other spacecraft, AXAF-S (Spectrometry) will be 'arched 
aboard a Delta rocket, possibly to a 1 0K by 100K km polar orbit. AXAF-I is currently approac g 
Systems Requirements Review. AXAF-S is still in the definition phase. 

Objective* Neither AXAF spacecraft are being designed for on-orbit maintenance, as the Hubble Space 
TelescoDe'was Thus the scope of potential VR applications is somewhat reduced. That is, operations 
TOtmSslor maintenance 8 and associated hardware features (e.g., restraints and mobility aids) am not 
beina considered. These areas provide a fertile environment for VR applications. However, since the 
AXAF I is shuttle-deployed there will be, at some point, planning for mission-success contingency 
E^eWaf^ can contribute to this process. Thmughout this pmcess analybcal 

studies compare alternate candidate design solutions and evaluate the consequences of design 
decisions. Operations development can be accomplished hand-in-hand with har ^ r ®^® lo( ^f nt ’ 
both in turn being refined during each iteration to approach an "optimal" man-machine > system^ This 
hardware and operations development can benefit through the application of VR technologies and 
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operations will benefit. 

VR Application/ Approach: Human Factors issues and considerations in hardware! and operations 
development present a large class of potential VR applications. VR technologies and techniques 
currently provide some limited macro- and micro-ergonomic analytical tools for c °^'^ '° t 0 
operational, viewing and reach envelope requirements, in both one-gravity and microgravity 

environments. 

An ainnrithm has been developed to rescale user anthropometric attributes to any desired virtual 
al SmSrJ a^th pScemne male could view and reach as a virtual 5th percentile Temale -and 

vice r s a Furt h e r a technique has been developed where the user can manipulate the attitude of virtual 

spacecraft’ as a whole, while "grabbing" a handrail, giving the egocentric perception of microgravrty 
mobility. This can provide some of the various advantages and disadvantages of reaching and 
maneuvering in microgravity. 

Combined with scaleable user anthropometry attributes, macro-ergonomics analyses for the topofagicaf 
desi^^fworif areas can consider what one is able to see from a variety of 

3&&g3sgSS5SSSg3£ 

able to touch from a variety of shoulder and seat reference points and/or foot restraint locations. 

VR technologies and techniques can be applied in the AXAF Program Many analyses 

toolkit. 
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In some instances, VR might be considered tor use in an analysis that would have otherwise not be 
undertaken. Resources (time, people, materials, etc.) required for a "standard" simulation or mock-up 
analysis may be greater than the expected return. In this case VR, due to its relatively low utilization costs, 
would surpass the cost/benefit ratio threshold and enable an analysis that would have otherwise been 
torgone. 

Similarly, VR can enhance and enable more effective utilization of standard simulations and mock-up 
analyses' By preceding these analyses with preliminary VR analyses, both the hardware and operations 
can be refined so that the return from the standard analyses is increased. This is accomplished by either 
reducing the magnitude or number of standard analyses and/or improving the fidelity of those analyses 
with a more mature design. 

Because the Virtual Worlds (VWs) are nothing more than computer files, design changes can be done 
more quickly and more candidate configurations can be subsequently analyzed than is currently possible 
with existing, "standard" Human Factor tools (e.g., Fomecor mockups). 

In addition to the planning and development of contingency EVAs, design of ground support and 
processing facilities and operations can also benefit through VR utilization. Using VR to visualize and 
interact with various AXAF control center configuration options, for example, provides a design analytical 
capability that is not otherwise possible. A more general operations and utilization application is related to 
initial AXAF ground support personnel training. As a member is assigned to the AXAF cadre there is a 
familiarization phase for AXAF systems and capabilities. 

There is a period during which the newcomer must quickly assimilate a large amount of information Into his 
or her concurrently evolving schema or mental model of AXAF. A tour through a virtual AXAF may initialize 
the newcomer and provide insights into system functionality and capabilities. If successful, this could 
provide a basis for a more accelerated training program and a better integrated understanding of AXAF 
systems. 

A virtual AXAF would enable a more convenient, even remote, method to "visualize" AXAF systems 
Updated VW files could even be electronically transmitted to other VR sites for visualizing. This ability to 
electronically transfer Virtual Worlds (VWs) further enhances the familiarization/initialization training 
application discussed above. In fact, another existing VR technology capability can enhance many AXAF 
applications. Using both the MSFC and JSC VR systems simultaneously, the users could enter and 
review contingency EVA procedures and hardware implications. 

Two major enhancements are required in object behavior and dynamics attributes for more advanced 
applications of VR technology. These include the incorporation of an anthropometric model and a physics 
properties simulator reflecting physical laws concerning motion and collisions. The former should inckide 
link lengths to reflect a broad anthropometric design range (e.g., 5th percentile female through the 95th 
percentile male) and realistic joint range-of-motion capabilities and constraints. This enhancement would 
enable dynamic work envelope analyses.The physics properties simulator should include realistic linear 
and angular acceleration/velocity and kinetic energy transfer. 

As these enhancements come on line, the training application discussed above, for example, can be 
expanded to cover more demanding training objectives. However, instead of discussing an expanded 
training application, another application will be proposed, although the discussion applies to the training 
application as well. 

A demanding and comprehensive application for VR is support of unplanned contingency EVAs. That Is. 
subsets of the features and VR capabilities required to support this application are used in a vanety ot 
other applications. Support to unplanned EVAs requires Human Factors analyses (e.g., viewing, reacn, 
and dynamic work envelope analyses), operations development, training, mission support, and even 
simultaneous participation by physically separated users in the same VE. 

It is anticipated that enhanced VR will be capable of supporting many of the activities and analyses that 
occur on the ground in support of an unplanned EVA. Viewing analyses, reach envelope analyses, and, 
with an incorporated anthropometric model, dynamic work envelope analyses can be achieved 
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concurrently with procedure development. Although much of this can be done in an engineering mock- 
up, VR offers several unique capabilities. 

First, VR could provide a timely and safe method to enable the various advantages and disadvantages of 
reaching and maneuvering in a microgravity environment. This includes body attitudes and positions 
difficult to recreate in a one-G environment. This would be superior to existing methods for simulating 
micro-gravity because existing methods can not be used in a timely manner and are of limited durations 
(KC-135) or require ancillary equipment (Neutral Buoyancy Simulator) that can interfere with operations in 
restricted volumes.Second, VR would permit anthropometric sizing to reflect the dimensions of the on- 
board crew. This is particularly useful for operations being planned in relatively tight spaces. 

VR would offer the mission and payload managers the ability to visualize the procedure and environment 
to qain a faster and more indepth understanding of the operation. This could be accomplished while the 
managers are sitting at their consoles in the control center. Further, managers^ at both centers, could 
enter the VW, simultaneously, to review and discuss the operation. This capability for direct mission 
support would be unprecedented, though the possibilities are not limited unplanned EVA. 

Benefits: The potential benefits from the application of VR technologies and techniques to AXAF are 
significant. More efficient utilization of constrained resources can be realized. 

Viewing analyses, reach envelope analyses, and, with an incorporated anthropometric model, dynamic 
work envelope analyses can be achieved concurrently with procedure development. VR can provide a 
timely and safe method to enable the various advantages and disadvantages of reaching and 
maneuvering in a microgravity environment. This would be superior to existing methods for simulating 
micro-gravity because existing methods can not be used in a timely manner and are of limited durations. 
Even where the KC-135 and/or the NBS are appropriate, prior utilization of virtual mockups can result in 
more efficient use of these micro-gravity simulators. Hardware and operations design can be more mature, 
resulting in fewer and/or more productive simulator sessions. 

Pre-mission operations development and validation can utilize VR. Pre-mission crew and cadre training 
could use the same VWs developed to support procedure development. This would prove particularly 
beneficial for operations where the various advantages and disadvantages of reaching and maneuvenng 
in a microgravity environment make a difference. 

A virtual AXAF would enable a more convenient, even remote, method to "visualize" AXAF systems 
Updated VW files could even be electronically transmitted to other VR sites for visualizing. 

VR can support operations development and validation during the deployment mission. VR would offer 
the mission and payload managers the ability to visualize the procedure and environment. Both could 
enter the VW, simultaneously, to review, discuss, and approve the operation. This capability for direct 
mission support would be unprecedented. 

Prepared by: 

Joseph P. Hale 

Man/Systems Integration Branch 
NASA Marshall Space Flight Center 
Marshall Space Flight Center, AL 35812 
(205) 544-2193 
jphale@ nasamail. nasa.gov 



Chapters: Conclusions 


General Issues 

• Visualization and spatial interpretation of massive databases, such a lunar 

/planetary surface, the human body, ocean-land interfaces, and so forth. 

• Interactive presence to manipulate, recombine, or restructure complex, 

environmental data sets. 

• General of visual, aural, or haptic representations via Information processing 

subsystems. 

• Degree of sensory distortion, imaging limitations and information 

representations congruent to normative human behavior. 

• Advanced visual, aural, and haptic rendering and feedback capabilities for 

analysis of real of hypothetical databases. 

• Advance computer science for data access, data processing, and multi- 

sensory fusion algorithms. 

• Advance US competitiveness and productivity in civilian and military 

applications, such as: 

Mission/event simulation for rehearsal and training 
Architectural layout for design and marketing 
Telepresence and teleoperations 
Public awareness 

Design and engineering development 
Medical, scientific and arts education 
Recreational and motivational enhancement 


NASA Issues 

A second set of issues will drive NASA's Virtual Environment technology are 
collectively called "Programmatic Issues". Some are Internal to NASA and 
reflect the expected direction of the Agency, future budget, and continued major 
programs and responsibilities of the agency-Space Transport System, Space 
Station, and Aeronautical programs that may benefit from VE technology. 

Other issues are external to NASA, but will impact the extent to which NASA 
takes a leadership role. These include: 

• National policy on VE as set by the President's Science Advisor, OSTP, and 

FCCSET. 

• Development of supporting technology in VE by other government agencies 

and private industry. 

• Marketplace forces for low-cost technology for mass market applications. 
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Center Activities in VE 


Ames Research Center 

* Responsible for human performance research relevant to developing VE for 

NASA applications. L , 

• Responsible for the development of human centered technology for 

aeronautics. 


Goddard Space Flight Center 

. Responsible for unmanned scientific studies and applications for unmanned 
space flight, in the areas of: 

Space physics 
Astrophysics 
Earth sciences 
Flight project support 

Jet Propulsion Laboratory .. . 

• Responsible for research, development and applications for unmanned 

spacecraft, satellites and ground data systems. 

• Responsible for manned space flight research, development, and applications. 

• Responsible for astronaut training. 

Marshall Space Flight Center 

• Responsible for spacecraft design, structure, development and operations. 

Conclusions 

Since beginning research and technology development in 1985, NASA Centers 
have learned important lessons about the technology itself and the value it can 
provide in accomplishing the gamut of NASA’s missions in aeronautics, science, 

and space. 

1. Cost savings could be dramatic since Virtual Environment can potentially 

allow change to be made in a small way which can have a large effect; can 
potentially analyze situations with Virtual Environment with capabilities not 
heretofore available; can potentially analyze situations quicker and cheaper 
than with conventional methods; analyses can potentially be done which 
allow unique insights for investigators/scientists. 

2. Networking is critically important to users of Virtual Environment because of 

the need to share data among many investigators. 

3 Since model and database development are critical and time consuming for 
virtual world development, techniques for streaming this modeling are 
essential. Standardization, and maintenance are also critical and need to 

be address. 
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4. NASA recognizes the need for human performance validation and that human 

performance requirements drive the technology. 

5. The productivity benefits of Virtual Environment will critically depend upon 

validated modeling of the specific task domain. 

6. Challenging mission applications within NASA call for a responsive Virtual 

Environment technology. Typically, this is a high technology need. NASA 
has a leadership role in the technology development without depending 
upon the value of low-tech commercial development. 

7. Virtual Environment is pervasive and the implications are extensive within 

NASA's many missions and research programs. NASA should be prepared 
to respond to such demands by supporting the technology. 

8. Uses of Virtual Environment technology for human performance applied 

studies and critical descriptive research matches both applied mission 
needs as well as fundamental research needs. 

9 Uses of Virtual Environment technology provide a flexible, relatively low-cost 
method for operational analysis, scientific studies, and critical discipline 
research. 

10. Although Virtual Environment technology Is evolutionary, building upon 

technologies such as simulation, computer graphics and so forth; 
implications for its use are revolutionary. 

11. A well-documented, international interest and economic position of Virtual 

Environment technology exists. NASA has a well understood role in 
technology development and transfer. This transfer must be fostered if the 
US is to maintain its leadership position. 

12. Current Virtual Environment systems generally do not have sufficient 

sensory-motor fidelity and human-machine interface design to deliver the 
performance necessary to achieve many of the above potential . 
applications but foreseeable technical advances may change this situation 
within 1-3 years. 
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